Attachable matter detection apparatus and attachable matter detection method

ABSTRACT

An attachable matter detection apparatus according to an embodiment includes an acquirement unit, a creation unit, and a determination unit. The acquirement unit acquires a determination target area of an attachable matter from a photographic image. The creation unit creates histograms of at least an edge intensity, luminance, and saturation for the determination target area acquired by the acquirement unit. The determination unit determines whether or not the attachable matter exists in the determination target area on the basis of a ratio of frequency of each grade in each of the histograms created by the creation unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2016-208071, filed on Oct. 24,2016; Japanese Patent Application No. 2016-208072, filed on Oct. 24,2016; and Japanese Patent Application No. 2017-031488, filed on Feb. 22,2017, the entire contents of all of which are incorporated herein byreference.

FIELD

Embodiments described herein relate to an attachable matter detectionapparatus and an attachable matter detection method.

BACKGROUND

In the related art, there is known an in-vehicle camera mounted on avehicle to photograph surroundings of the vehicle. An image photographedby the in-vehicle camera is monitored and displayed, for example, toassist driver's visibility and is used in sensing to detect a white lineon the road or an approaching object to the vehicle.

Incidentally, an attachable matter such as raindrops, snowflakes, dust,and mud is attached to a lens of the in-vehicle camera and may hinderthe visibility assistance or the sensing described above. In thisregard, a technique of removing an attachable matter by spraying washingwater or compressed air to the lens of the in-vehicle camera has beenproposed. In this technique, for example, a detection algorithm fordetecting an attachable matter on a lens by analyzing a photographedimage of the in-vehicle camera may be employed (for example, seeJapanese Laid-open Patent Publication No. 2001-141838).

However, in the related art described above, there is a need forimprovement in terms of improving accuracy of detecting an attachablematter.

The aforementioned detection algorithm includes, for example, detectingan edge from a photographed image and extracting a contour of theattachable matter on the basis of such edges. However, an image of theattachable matter such as a raindrop may blur, and the contour may blurin some cases. Therefore, it was difficult to perform detection withhigh accuracy in some cases.

Even when the contour of the raindrop is clear, for example, a structurehaving a shape similar to the raindrop may be erroneously detected as araindrop.

SUMMARY

An attachable matter detection apparatus according to an aspect of anembodiment includes an acquirement unit, a creation unit, and adetermination unit. The acquirement unit configured to acquire adetermination target area of an attachable matter from a photographicimage. The creation unit configured to create histograms of at least anedge intensity, luminance, and saturation for the determination targetarea acquired by the acquirement unit. The determination unit configuredto determine whether or not the attachable matter exists in thedetermination target area on the basis of a ratio of frequency of eachgrade in each of the histograms created by the creation unit.

BRIEF DESCRIPTION OF DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIGS. 1A to 1D are (first to fourth) explanatory diagrams schematicallyillustrating an attachable matter detection method according to anembodiment;

FIG. 2A is a schematic diagram for describing a first embodiment;

FIG. 2B is a schematic diagram for describing a second embodiment;

FIG. 3 is a block diagram illustrating an attachable matter removalsystem according to the first embodiment;

FIGS. 4A to 4C are (first to third) diagrams illustrating a specificexample of an exclusion condition;

FIGS. 5A to 5C are (first to third) explanatory diagrams illustrating amodification of the exclusion condition;

FIG. 6 is a diagram illustrating a specific example of adjustment of theexclusion condition;

FIG. 7 is a flowchart illustrating a processing sequence executed by theattachable matter detection apparatus according to the first embodiment;

FIGS. 8A to 8D are (first to fourth) diagrams illustrating an example ofsetting partitioned areas;

FIG. 9 is a diagram illustrating a specific example of storedinformation according to the second embodiment;

FIG. 10 is a diagram illustrating a specific example of a detectioncondition;

FIG. 11 is a diagram illustrating an exemplary parameter setting screen;

FIGS. 12A and 12B are (first and second) diagrams illustrating aspecific example of a determination method in a case where the previouschange amounts one and two versions ago are included;

FIG. 13 is a diagram illustrating a specific example of adjustment ofthe detection condition;

FIG. 14 is a flowchart illustrating a processing sequence executed by anattachable matter detection apparatus according to the secondembodiment;

FIG. 15 is a block diagram illustrating an attachable matter removalsystem according to a third embodiment;

FIG. 16A is a diagram illustrating an exemplary content of notificationfrom an attachable matter detector;

FIG. 16B is a diagram illustrating an exemplary content of dataregarding a detection area included in a detection information DB;

FIG. 16C is an explanatory diagram illustrating a detection area state;

FIGS. 17A to 17D are (first to fourth) processing explanatory diagramsof an inter-algorithm overlap determination unit;

FIGS. 18A to 18C are (first to third) processing explanatory diagrams ofan inter-frame overlap determination unit;

FIG. 18D is a processing explanatory diagram of an inter-frame overlapdetermination unit and an attachment determination unit;

FIG. 18E is a processing explanatory diagram of a removal necessitydetermination unit;

FIG. 19 is a flowchart illustrating a processing sequence executed by anattachable matter removal system according to the third embodiment;

FIG. 20 is a diagram illustrating an overview of an attachable matterdetection method according to fourth to sixth embodiments;

FIG. 21 is a block diagram illustrating an attachable matter detectionapparatus according to the fourth embodiment;

FIG. 22 is a diagram illustrating a binarization threshold value;

FIG. 23 is a diagram illustrating an exemplary template according to thefourth embodiment;

FIG. 24 is a diagram illustrating a template scanning position;

FIG. 25 is a diagram illustrating an exemplary matching process of amatching unit;

FIGS. 26A to 26D are (first to fourth) diagrams illustrating a detectionprocess of a detection unit;

FIG. 27 is a flowchart illustrating a processing sequence executed by anattachable matter detection apparatus according to the fourthembodiment;

FIG. 28 is a block diagram illustrating an attachable matter detectionapparatus according to the fifth embodiment;

FIG. 29 is a diagram illustrating an extraction range;

FIG. 30A is a diagram illustrating a vector calculation method;

FIG. 30B is a diagram illustrating exemplary parameter information;

FIG. 31 is a diagram illustrating an exemplary template according to thefifth embodiment;

FIG. 32 is a diagram illustrating a detection threshold value accordingto the fifth embodiment;

FIG. 33 is a flowchart illustrating a processing sequence executed by anattachable matter detection apparatus according to the fifth embodiment;

FIG. 34 is a block diagram illustrating an attachable matter detectionapparatus according to the sixth embodiment;

FIGS. 35A and 35B are (first and second) diagrams for describing amethod of calculating a representative value;

FIG. 36A is a diagram illustrating an exemplary template according tothe sixth embodiment;

FIG. 36B is a diagram illustrating an exemplary matching process of thematching unit according to the sixth embodiment;

FIG. 37 is a diagram illustrating a detection process using a detectionunit according to the sixth embodiment;

FIG. 38 is a diagram illustrating an exclusion example of the detectionprocess using the detection unit according to the sixth embodiment; and

FIG. 39 is a flowchart illustrating a processing sequence executed bythe attachable matter detection apparatus according to the sixthembodiment.

DESCRIPTION OF EMBODIMENTS

An attachable matter detection apparatus and an attachable matterdetection method according to an embodiment of the present applicationwill now be described in details with reference to the accompanyingdrawings. The present disclosure is not limited to the embodimentdescribed in the following.

In the following description, an overview of the attachable matterdetection method according to this embodiment will be described withreference to FIGS. 1A to 1D. Then, an attachable matter detectionapparatus 10 obtained by applying the attachable matter detection methodaccording to the embodiment will be described with reference to FIG. 2Aand subsequent figures.

First, an overview of the attachable matter detection method accordingto this embodiment will be described with reference to FIGS. 1A to 1D.FIGS. 1A to 1D are (first to fourth) explanatory diagrams illustratingan overview of the attachable matter detection method according to anembodiment.

As illustrated in FIG. 1A, for example, in-vehicle cameras such as afront camera 2-1, a rear camera 2-2, a right-side camera 2-3, and aleft-side camera 2-4 are mounted on a vehicle C to photographsurroundings of the vehicle C. Note that, in the following description,such in-vehicle cameras will be collectively referred to as a “camera2.”

The camera 2 has an image sensor such as a charge coupled device (CCD)or a complementary metal oxide semiconductor (CMOS) to photographsurroundings of the vehicle C using such an image sensor. In addition,the camera 2 outputs the photographed image, for example, to anattachable matter removal system 1 including the attachable matterdetection apparatus 10 according to this embodiment.

Note that a wide-angle lens such as a fisheye lens is employed in a lens2 a of the camera 2 (refer to FIG. 1B), and each camera 2 has an angleview of 180° or larger. By using them, it is possible to photograph theentire circumference of the vehicle C.

As illustrated in FIG. 1B, the attachable matter removal system 1according to this embodiment has an attachable matter remover 3 thatremoves an attachable matter such as raindrops, snowflakes, dust, andmud attached on the lens 2 a of the camera 2.

The attachable matter remover 3 has a nozzle 3 a. The nozzle 3 a has aninjection port provided toward the lens 2 a to remove attachable matter,for example, by injecting compressed air supplied through a compressedair supply source 3 b and a valve 3 c and a washing liquid suppliedthrough a washing liquid supply source 3 d and a valve 3 e toward thelens 2 a.

Note that an operation control of the attachable matter remover 3 isperformed by a removal determination device 5 provided in the attachablematter removal system 1. The removal determination device 5automatically determines whether or not attachable matter is attached onthe lens 2 a, and whether or not it is necessary to remove theattachable matter on the basis of a detection result of the attachablematter detection apparatus 10. If it is necessary to remove theattachable matter, the removal determination device 5 allows theattachable matter remover 3 to perform a removal operation.

In the attachable matter detection apparatus 10 according to thisembodiment, in order to improve attachable matter detection accuracycontributing to such automatic determination, at least an edge intensityhistogram, a luminance histogram, and a saturation histogram of eachpixel are created for a determination target area for determiningwhether or not there is an attachable matter, and the attachable matteris determined on the basis of frequencies of each grade of thehistograms.

Specifically, as illustrated in FIG. 1C, the attachable matter detectionapparatus 10 acquires a detection area which is an area where existenceof the attachable matter is estimated and is detected from the cameraimage of the camera 2, for example, using a plurality of detectionalgorithms (Step S1).

The attachable matter detection apparatus 10 creates histograms of theedge intensity, the luminance, and the saturation of the acquireddetection area, for example, by classifying them into three gradesincluding “low,” “middle,” and “high” grades (Step S2). Note that aspecific example of the method of obtaining the edge intensity, theluminance, and the saturation will be described in conjunction with ahistogram creation unit 11 b (described below)

The attachable matter detection apparatus 10 determines whether or notthe attachable matter estimated to exist in the detection area is trulyan attachable matter on the basis of a “ratio” between frequencies ofeach grade of each created histogram (Step S3). Specifically, if such aratio of the frequencies satisfies a predetermined exclusion conditionfor excluding the attachable matter, it is determined that theattachable matter estimated from the detection area is not an attachablematter.

For example, FIG. 1C illustrates a case where a tire portion of anothervehicle is detected as the detection area. However, the attachablematter detection apparatus 10 creates the histogram for such a detectionarea and compares the “ratio” of each frequency of each histogram with apredetermined exclusion condition.

As a result, since the detection area as the tire portion satisfies thepredetermined exclusion condition, the attachable matter detectionapparatus 10 determines that the attachable matter estimated from such adetection area is not an attachable matter. Note that a specific exampleof the exclusion condition will be described below with reference toFIGS. 4A to 4C and the like.

The attachable matter detection apparatus 10 considers the detectionarea determined as not the attachable matter as an erroneous detectionpart and excludes the detection area from a processing target of theremoval determination device 5 of the subsequent stage. That is, theattachable matter detection apparatus 10 does not notify the removaldetermination device 5 of such a detection area.

As a result, it is possible to reduce a processing load of the removaldetermination process in the removal determination device 5. Inaddition, by performing an error detection determination process of theattachable matter detection apparatus 10 for the detection area detectedusing a plurality of detection algorithms, it is possible to assisterror detection of each detection algorithm and contribute to improvingthe attachable matter detection accuracy.

Note that, here, for example, the edge intensity, the luminance, and thesaturation are selected from elements of the detection area. However,other elements such as a color or a standard deviation may also beemployed, and the element serving as a histogram creation target is notlimited. In addition, the grade is not limited to three grades including“low,” “middle,” and “high.”

However, although a case where the attachable matter detection apparatus10 is in an assistant position of each detection algorithm has beendescribed by way of example in FIG. 1C, the attachable matter detectionapparatus 10 may be configured to execute one of the detectionalgorithms.

In this case, specifically, as illustrated in FIG. 1D, the attachablematter detection apparatus 10 sets a plurality of partitioned areas fora single frame of the camera image of the camera 2 and acquires each ofthe partitioned areas (Step S1′).

The attachable matter detection apparatus 10 creates each histogram ofthe edge intensity, the luminance, and the saturation of each acquiredpartitioned area (Step S2′). Information including such histograms isstored for the previous frame one or more versions ago as well as thecurrent frame.

The attachable matter detection apparatus 10 determines whether or notthe attachable matter exists in each partitioned area on the basis of a“change amount” between the current frame and the previous frame (StepS3′). Note that FIG. 1D illustrates an example of the edge intensity.The attachable matter detection apparatus 10 determines whether or notthe attachable matter exists, for example, on the basis of tendencyindicated by a history of the current change amount of the edgeintensity, the previous change amount one version ago, and the previouschange amount two versions ago.

By performing determination on the basis of the “change amount” betweenframes in this manner, for example, even in a case of the raindrophaving a blur contour by which it is difficult to detect an edge, it ispossible to easily detect the raindrop as the attachable matter on thebasis of the characteristic of the raindrop indicated by the “changeamount.” That is, it is possible to contribute to improving theattachable matter detection accuracy. In addition, by performingdetermination for each partitioned area, it is possible to detect theattachable matter with a higher resolution, compared to a case wheredetermination is performed for all of the frames. Therefore, it ispossible to contribute to improving the attachable matter detectionaccuracy.

Such a detection condition based on the “change amount” may be set foreach partitioned area. As a result, it is possible to perform suitabledetermination depending on the characteristic of each partitioned area,for example, easiness of appearance of the change.

The detection condition, the processing, or the like when the “changeamount” is employed in this manner will be described below in moredetails with reference to FIGS. 9 to 13 and the like.

An attachable matter detection apparatus 10 according to an embodimentobtained by applying the attachable matter detection method describedabove will now be described in more details.

Note that, in the following description, it is assumed that the exampleof FIG. 1C is the first embodiment, and the example of FIG. 1D is thesecond embodiment. However, for easy description purposes, the first andsecond embodiments will be described in summary with reference to FIGS.2A and 2B.

As illustrated in FIG. 2A, in the description of the first embodiment,it is assumed that the attachable matter detection apparatus 10 has aconfiguration for performing an error detection determination process(for excluding erroneous detection parts) by using an externalattachable matter detection apparatus 4 that executes the attachablematter detection algorithms-1, -2, . . . , and -n as a front stage(refer to FIG. 3) and using a removal determination device 5 thatexecutes the removal determination process as a rear stage (refer toFIG. 1C).

As illustrated in FIG. 2B, in the description of the second embodiment,it is assumed that the attachable matter detection apparatus 10 has aconfiguration for executing one of the attachable matter detectionalgorithms-1, -2, . . . , and -n (refer to FIG. 1D). Note that the errordetection determination process indicated by the dotted line may beperformed by the configuration of the first embodiment or may beomitted.

First Embodiment

FIG. 3 is a block diagram illustrating the attachable matter removalsystem 1 according to the first embodiment. Note that, in FIG. 3, onlyelements necessary to describe the characteristics of this embodimentare illustrated as functional blocks, and general elements are notillustrated for simplicity purposes.

In other words, each element illustrated in FIG. 3 is just functionaland conceptual, and is not necessarily configured as illustrated in aphysical sense. For example, a distributed and/or integrated version ofeach functional block is not limited to those illustrated, and itsentirety or a part thereof may be functionally or physically distributedor integrated in an arbitrary unit depending on various loads, usesituations, and the like.

As illustrated in FIG. 3, the attachable matter removal system 1includes a camera 2, an attachable matter remover 3, one or moreexternal attachable matter detection apparatuses 4 (for example,external attachable matter detection apparatuses 4-1, 4-2, 4-3, . . . ),a removal determination device 5, and an attachable matter detectionapparatus 10.

Note that, as illustrated in FIG. 3, the external attachable matterdetection apparatus 4, the attachable matter detection apparatus 10, andthe removal determination device 5 may be used as elements of a removalcontrol device 50 that controls the entire process from detection of anattachable matter to removal of the attachable matter. A specificconfiguration of the removal control device 50 will be described belowin conjunction with the third embodiment following this embodiment andthe second embodiment.

Since the camera 2 and the attachable matter remover 3 have beendescribed above, they will not be described here. The externalattachable matter detection apparatus 4 acquires a camera image from thecamera 2 in a frame-by-frame manner, extracts a detection area estimatedas having an attachable matter from the camera image using a detectionalgorithm associated with each camera image, and notifies the attachablematter detection apparatus 10 of the extracted detection area.

As described above, the removal determination device 5 allows theattachable matter remover 3 to perform a removal operation on the basisof the detection result of the attachable matter detection apparatus 10when it is necessary to remove the attachable matter.

The attachable matter detection apparatus 10 has a control unit 11 and amemory unit 12. The control unit 11 includes a target area acquirementunit 11 a, a histogram creation unit 11 b, an attachable matterdetermination unit 11 c, and a condition adjustment unit 11 d.

The memory unit 12 is a storage device such as a hard disk drive, anonvolatile memory, or a register and stores condition information 12 a.

The control unit 11 controls the entire attachable matter detectionapparatus 10. The target area acquirement unit 11 a acquires thedetection area notification of which is provided from the externalattachable matter detection apparatus 4 as a determination target area.Note that, in the second embodiment described below, the target areaacquirement unit 11 a acquires the partitioned area from the cameraimage of the camera 2 as the determination target area.

The histogram creation unit 11 b creates at least the edge intensityhistogram, the luminance histogram, and the saturation histogram of eachpixel for each detection area acquired by the target area acquirementunit 11 a as many as a predetermined number of grades classified inadvance. The predetermined number of grades is set to three gradesincluding, for example, “low,” “middle,” and “high” as described above.

Specifically, for the edge intensity, the histogram creation unit 11 bconverts the image of the detection area into a grayscale image byperforming grayscale conversion. Note that the grayscale conversionrefers to a conversion process of expressing each pixel of the cameraimage in gray scales from white to black depending on the luminance.

The histogram creation unit 11 b extracts edge information of each pixelin the grayscale image by applying a Sobel filter to the grayscaleimage. Here, the edge information refers to the edge intensity in theX-axis direction and the Y-axis direction of each pixel. Note that,instead of the Sobel filter, another edge extraction method such as aLaplacian filter may also be employed.

The histogram creation unit 11 b calculates the edge intensity as arepresentative value of each pixel of the grayscale image on the basisof the extracted edge information of each pixel. Specifically, a valueobtained by squaring each of the edge intensities of the X-axis andY-axis directions as the edge information and adding them is calculatedas a representative value of the edge intensity of each pixel.

The histogram creation unit 11 b creates the edge intensity histogram onthe basis of the calculated representative value of the edge intensityof each pixel. Specifically, the histogram creation unit 11 b normalizeseach calculated representative value of the edge intensity to a value,for example, between 0 and 1. If three grades including “low,” “middle,”and “high” are set as described above, the histogram is created, forexample, by setting a value equal to or greater than 0 and smaller than0.3 to the “low” grade, setting a value equal to or greater than 0.3 andsmaller than 0.7 to the “middle” grade, and setting a value equal to orgreater than 0.7 and equal to or smaller than 1 to the “high” grade.

For the luminance, the histogram creation unit 11 b uses the luminanceof each pixel calculated on the basis of RGB values of each pixel (R: 0to 255, G: 0 to 255, and B: 0 to 255) in the grayscale conversiondescribed above. For example, the histogram creation unit 11 b mayextract only one element value out of R, G, and B of each pixel as arepresentative value and use this value as the luminance of each pixel.

For example, the histogram creation unit 11 b may calculate a simpleaverage value of each element value of R, G, and B as the luminance. Inaddition, for example, the histogram creation unit 11 b may calculatethe average value weighted using a so-called national television systemcommittee (NTSC) weighted average method or the like based on a formula“luminance=0.298912×R+0.586611×G+0.114478×B” as the luminance.

The histogram creation unit 11 b creates the luminance histogram on thebasis of the calculated luminance of each pixel. Normalization of theluminance or a classification method for a predetermined number ofgrades is similar to that of the edge intensity.

For the saturation, for example, in the case of the HSV color space, thehistogram creation unit 11 b may calculate the saturation on the basisof formulas “brightness (V)=Imax, and saturation (S)=(Imax−Imin)/Imax”where “Imax” refers to a maximum value of R, G, and B, and “Imin” refersto a minimum value.

For example, in the case of the HSL color space, the histogram creationunit 11 b may calculate the saturation on the basis of formulas“brightness (L)=(Imax+Imin)/2,” “saturation (S)=(Imax−Imin)/(Imax+Imin)when L≤0.5,” and “saturation (S)=(Imax−Imin)/(2−Imax−Imin) when L>0.5.”Note that the brightness (V) or the brightness (L) may also be used asthe aforementioned luminance.

The histogram creation unit 11 b creates the saturation histogram on thebasis of the calculated saturation of each pixel. Normalization of thesaturation or a classification method for a predetermined number ofgrades is similar to those of the edge intensity and the luminance.

Note that the histogram creation unit 11 b preferably enlarges orreduces the size of the detection area to match a standard size when thehistogram is created. As a result, it is possible to suppress avariation in detection accuracy caused by a difference in size bycomparing a ratio of the frequency of each element in the detection areain which the size is considered to be different with the predeterminedexclusion condition. That is, it is possible to contribute to improvingthe attachable matter detection accuracy.

The attachable matter determination unit 11 c determines whether or notthe attachable matter estimated to exist in the detection area is trulyan attachable matter on the basis of the ratio of the frequency of eachgrade of each histogram created by the histogram creation unit 11 b incomparison with a predetermined exclusion condition. The predeterminedexclusion condition is a combination of the ratios for each of threegrades (“low,” “middle,” and “high”) for each of the edge intensity, theluminance, and the saturation so as not to match the characteristics ofthe attachable matter and is contained in the condition information 12 ain advance.

Here, FIGS. 4A to 4C illustrate a specific example of the predeterminedexclusion condition. FIGS. 4A to 4C are (first to third) diagramsillustrating a specific example of the exclusion condition.

As illustrated in FIG. 4A, for example, the “exclusion condition-1” maybe set such that the “high” of the “edge intensity” is 50% or higher,the “low” of the “luminance” is 30% or higher, and “high” of the“saturation” is 50% or higher.

As illustrated in FIG. 4B, for example, the “exclusion condition-2” maybe set such that the “high” of the “edge intensity” is 50% or higher,the “middle” of the “luminance” is 50% or higher, and “low” of the“saturation” is 50% or higher.

As illustrated in FIG. 4C, for example, the “exclusion condition-3” maybe set such that the “middle” of the “luminance” is 80% or higher, andthe “high” of the “saturation” is 80% or higher.

All of the exclusion conditions of FIGS. 4A to 4C are conditions notsuitable for the characteristics of raindrops obtained through averification test or the like. Using the exclusion conditions of FIGS.4A to 4C, it is possible to contribute to improving the detectionaccuracy, particularly, of raindrops out of attachable matters.

Returning to FIG. 3, the attachable matter determination unit 11 cexcludes a detection area that satisfies such a predetermined exclusioncondition and is determined as not being an attachable matter from aprocess target of the rear stage. In addition, the attachable matterdetermination unit 11 c notifies the removal determination device 5 of adetection area that does not satisfy the predetermined exclusioncondition and is determined as an attachable matter in order to set itas a processing target of the rear stage.

Meanwhile, although the requirements of the exclusion condition include“edge intensity,” “luminance,” and “saturation” in the aforementionedexample, other requirements may also be added as described above.

Such a modification of the exclusion condition will be described withreference to FIGS. 5A to 5C. FIGS. 5A to 5C are (first to third)explanatory diagrams illustrating a modification of the exclusioncondition.

For example, the exclusion condition may include similarity or the likeobtained by matching a detection area and a predetermined template.

In such a modification, a vector direction of each pixel of thedetection area is focused. Specifically, as illustrated in FIG. 5A, insuch a modification, for example, the histogram creation unit 11 bcreates a vector direction image Vd by expressing the detection area Dain a vector direction image format.

More specifically, the histogram creation unit 11 b calculates vectordirections of each pixel from the detection area Da, creates a vectordirection image Vd in which the vector direction is colored, forexample, on the basis of a color wheel, and notifies the attachablematter determination unit 11 c of the created vector direction image Vd.

The attachable matter determination unit 11 c calculates, for example,cross-correlation coefficients for all target pixels by performingtemplate matching using the vector direction image Vd notification ofwhich is provided and a template Ti provided in advance on the basis ofthe color wheel and performs condition determination includingsimilarity indicated by such a cross-correlation coefficient.

Note that the template Ti illustrated in FIG. 5A shows, for example, acharacteristic of an inwardly bright raindrop. As illustrated in FIG.5B, for the template Ti, various variations such as a template Ti-1 in(a) of FIG. 5B or a template Ti-2 in (b) of FIG. 5B may be prepareddepending on a size or shape of the detection area Da, a characteristicon how the raindrop shines, and the like. In this case, it is possibleto deal with various shapes of the attachable matter.

In a case where the cross-correlation coefficient described above isemployed, the similarity is expressed within a range between −1 and 1.Therefore, for example, an exclusion condition of such a modificationincludes a condition “similarity<threshold value” or a condition“similarity<0” as a requirement of “+α” as illustrated in FIG. 5C.

In this manner, if the exclusion condition includes similarity of thetemplate matching result as well as the frequency ratio of thehistogram, it is possible to contribute to improving the attachablematter detection accuracy.

Returning to FIG. 3, in a case where a predetermined trigger suitablefor adjusting the exclusion condition is generated, the conditionadjustment unit 11 d adjusts the exclusion condition and performs aprocess of appropriately updating the condition information 12 a.

A specific example of this case will be described with reference to FIG.6. FIG. 6 is a diagram illustrating a specific example of adjustment ofthe exclusion condition. As illustrated in FIG. 6, as a trigger foradjusting the exclusion condition, for example, “scene change timing”may be employed.

The scene change timing corresponds to, for example, a case where achange of day and night or landscape that can be determined by analyzinga camera image from the camera 2 is detected.

In such a case, as an example of the adjustment content, for example, itis conceived that, when a scene change to an urban area at night isdetected, raindrops as an attachable matter are strongly reflected bymany light sources existing in streets, and their contours become clear(the ratio of “high” in the edge intensity increases). Therefore, as anexclusion condition for excluding raindrops in such a case, for example,adjustment may be performed by decreasing the ratio of the “high” in theedge intensity (refer to “⬇” in the drawings).

As another trigger for adjusting the exclusion condition, for example,the “detection area position” may be used. For example, if the detectionarea is a position corresponding to the sky, it is considered that thereis little influence on the visibility during driving of a driver.Therefore, in this case, for example, the condition adjustment unit 11 dperforms adjustment such that the detection area of this position isunconditionally excluded as an example of the adjustment content.

As further another trigger for adjusting the exclusion condition, forexample, “continuity between a plurality of frames” may be used. Forexample, if the detection area is detected in only a single frame, apossibility of the attachable matter is considered to be low. In thiscase, for example, the condition adjustment unit 11 d performsadjustment such that the detection area having a single frame isunconditionally excluded as an example of the adjustment content.

In this manner, if the exclusion condition is adjusted in a case where apredetermined trigger suitable for adjustment of the exclusion conditionis generated, it is possible to detect an attachable matter suitablydepending on a situation during driving of the vehicle C. That is, it ispossible to contribute to improving the attachable matter detectionaccuracy.

Next, a processing sequence executed by the attachable matter detectionapparatus 10 according to this embodiment will be described withreference to FIG. 7. FIG. 7 is a flowchart illustrating a processingsequence executed by the attachable matter detection apparatus 10according to the first embodiment.

First, the target area acquirement unit 11 a acquires a detection areaDa of each detection algorithm of the external attachable matterdetection apparatus 4 (Step S101). In addition, the histogram creationunit 11 b creates histograms for each of the edge intensity, theluminance, and the saturation of the detection area Da acquired by thetarget area acquirement unit 11 a (Step S102).

The attachable matter determination unit 11 c determines whether or notan attachable matter exists on the basis of the ratio of frequency ofeach grade of each histogram created by the histogram creation unit 11 b(Step S103).

Here, if it is determined that the detected object is not an attachablematter (Step S104, Yes), the attachable matter determination unit 11 cexcludes the corresponding detection area Da (Step S105). Otherwise, ifit is determined that the detected object is an attachable matter (StepS104, No), the attachable matter determination unit 11 c notifies theremoval determination device 5 of this detection area (Step S106).

The control unit 11 determines whether or not there is a processing endevent (Step S107). The processing end event corresponds to, for example,IG OFF, ACC OFF, and the like. Here, if it is determined that there isno processing end event (Step S107, No), the processing from Step S101is repeated. In addition, if it is determined that there is a processingend event (Step S107, Yes), the attachable matter detection apparatus 10terminates the processing.

As described above, the attachable matter detection apparatus 10according to the first embodiment includes the target area acquirementunit 11 a (corresponding to an example of “acquirement unit”), thehistogram creation unit 11 b (corresponding to an example of “creationunit”), and the attachable matter determination unit 11 c (correspondingto an example of “determination unit”).

The target area acquirement unit 11 a acquires a detection area Da of anattachable matter (corresponding to an example of “determination targetarea”) in a camera image (corresponding to an example of “photographicimage”). The histogram creation unit 11 b creates at least histograms ofthe edge intensity, the luminance, and the saturation for the detectionarea Da acquired by the target area acquirement unit 11 a. Theattachable matter determination unit 11 c determines whether or notthere is an attachable matter in the detection area Da on the basis ofthe ratio of the frequency of each grade of each of the histogramscreated by the histogram creation unit 11 b.

Using the attachable matter detection apparatus 10 according to thefirst embodiment, it is possible to improve the attachable matterdetection accuracy. In addition, since the unnecessary detection area Dacan be excluded from the processing target of the rear stage, it ispossible to achieve an effect of suppressing a processing load of theentire system.

Second Embodiment

Next, a second embodiment will be described with reference to FIGS. 3and 8A to 14. As described above, according to the second embodiment,the attachable matter detection apparatus 10 executes one of theattachable matter detection algorithms-1, -2, . . . , and -n (refer toFIG. 2B). In this case, it is determined whether or not there is anattachable matter on the basis of the “change amount” between frames ineach of the edge intensity histogram, the luminance histogram, and thesaturation histogram of each partitioned area for each partitioned area(refer to FIG. 1D).

Note that a block structure of the attachable matter detection apparatus10 according to the second embodiment can be described on the basis ofthe block diagram of the first embodiment. Therefore, for convenientdescription purposes, the description will be focused on the partsdifferent from those of the first embodiment using the block diagram ofFIG. 3 described above.

Referring to FIG. 3, in the attachable matter detection apparatus 10according to the second embodiment, the target area acquirement unit 11a sets a plurality of partitioned areas for a single frame of a cameraimage of the camera 2 and acquires each of these partitioned areas as adetermination target area.

Here, an exemplary setting of the partitioned area will be describedwith reference to FIGS. 8A to 8D. FIGS. 8A to 8D are (first to fourth)diagrams illustrating an exemplary setting of the partitioned area. Asillustrated in FIG. 8A, the partitioned area may be set by partitioningthe entire camera image of a single frame, for example, into ninepartitioned areas (3 by 3).

Note that, in the following description, it is assumed that the case ofFIG. 8A will be used as a predominant example, and upper, middle, andlower parts of a screen refer to upper, middle, and lower stagescorresponding to upper, middle, and lower areas, respectively, of thedrawing.

Although nine areas (3 by 3) are set in this case, the partitioningnumber is not limited, but may be set to sixteen areas (4 by 4). If thepartitioned areas are more reduced, it is possible to contribute toimproving detection accuracy of a small raindrop or the like.

As illustrated in FIG. 8B, the size of the partitioned area may be notequal to each other. For example, as illustrated in FIG. 8B, for an areasuch as the vicinity of the center, considered to significantlyinfluence on visibility of a driver when an attachable matter isattached, the partitioned area may be further reduced. As a result, itis possible to increase sensitivity for detecting the change amount foran area requiring high visibility for safety purposes or the like.

As illustrated in FIG. 8C, the partitioned area is not limited to arectangular shape, but may be set to, for example, a circular shapematching the shape of the raindrop depending on the shape of theattachable matter. As a result, it is possible to easily recognize thechange amount depending on the shape.

As illustrated in FIG. 8D, for example, a partitioned area determined asnot significantly influencing on driver's visibility may be excludedfrom the target area in advance. In addition, such an area excluded fromthe target may be set variably depending on a change of a drivingsituation, and the like.

The partitioning number or the like may be changed depending on a changeof the driving situation as well. For example, in day and night, it isconsidered that the change amount of the edge intensity, the luminance,or the like is smaller in the night. Therefore, when it is detected thatnight has come through image analysis or the like, sensitivity fordetecting the change amount may be improved by dynamically increasingthe partitioning number.

Returning to FIG. 3, in the attachable matter detection apparatus 10according to the second embodiment, the histogram creation unit 11 bcreates each of the edge intensity histogram, the luminance histogram,and the saturation histogram of each acquired partitioned area.Information containing such histograms is stored for a previous frameone or more versions ago as well as the current frame.

The stored information will be described with reference to FIG. 9 indetails. FIG. 9 is a diagram illustrating a specific example of thestored information according to the second embodiment. As illustrated inFIG. 9, according to the second embodiment, the created histograms arestored, for example, for the current frame, the previous frame oneversion ago, and the previous frame two versions ago. In addition, thechange amount calculated from such a histogram, or the like is alsostored.

More specifically, as illustrated in the stored information of FIG. 9,the stored information includes, at least, the number of data of thecurrent frame (each frequency), the numbers of data of the previousframes one and two versions ago, a simple moving average (SMA), a changeamount between the current frame and the simple moving average, and achange amount between the previous frames one and two versions ago andthe simple moving average.

Note that, in the drawings, the mark “*” indicates a solution of the SMAaccording to this embodiment, and the mark “**” indicates a solution ofthe change amount according to this embodiment.

Returning to FIG. 3, in the attachable matter detection apparatus 10according to the second embodiment, the attachable matter determinationunit 11 c at least compares the current change amount of each grade ofthe histogram and a progress of the previous change amount one versionago with a predetermined detection condition on the basis of the storedinformation described above, and determines that there is an attachablematter if such a detection condition is satisfied. The predetermineddetection condition is included in the condition information 12 a inadvance.

Here, a specific example of the predetermined detection condition isillustrated in FIG. 10. FIG. 10 is a diagram illustrating a specificexample of the detection condition.

As illustrated in FIG. 10, for example, the “detection condition”according to the second embodiment may include a case where the “high”of the “edge intensity” decreases by a predetermined amount, the “low”of the “edge intensity” increases by a predetermined amount, the“middle” of the “luminance” increases by a predetermined amount, and the“low” of the “saturation” increases by a predetermined amount.

A threshold value or the like for determining increase/decrease includedin such a detection condition may be set as a parameter in eachpartitioned area. For example, FIG. 11 is a diagram illustrating anexample of the parameter setting screen. Note that such a parametersetting screen is illustrated for convenient description purposes inorder to describe that parameters can be set for each partitioned area,and is not indispensable in the system.

As illustrated in FIG. 11, on the parameter setting screen, a widgetsuch as a checkbox or a slider is used for each element (“edgeintensity,” “luminance,” and “saturation”) of the histogram and eachgrade of these elements to allow setting of the threshold valuecorresponding to the “predetermined amount.”

For example, in the “edge intensity” column, the “high” and “low” boxesare checked, and the slider is controlled. In the case of “high,” acondition of “decrease” can be set by setting the threshold value to anegative value. In addition, in the case of “low,” a condition of“increase” can be set by setting the threshold value to a positivevalue.

In such a parameter setting screen, threshold values can be setindividually for areas corresponding to each of the “upper,” “middle,”and “lower” rows of the screen (refer to FIG. 8A). For example, in FIG.11, an example is illustrated in which an area corresponding to “lower”of a screen surrounded by a closed curve of a dotted line is setdifferently from that of the “upper” or “middle.”

As for the camera image from the camera 2 mounted on the vehicle C,since the change amount tends to be greater in the “lower” of thescreen, parameters are set individually depending on such acharacteristic. In this manner, by making it possible to set theparameter of the change amount depending on each characteristic of thepartitioned area, it is possible to contribute to improving theattachable matter detection accuracy.

In the attachable matter determination unit 11 c according to the secondembodiment, it is possible to improve the attachable matter detectionaccuracy by determining a progress including the previous change amounttwo versions ago in addition to the current change amount and theprevious change amount one version ago for each grade of the histogram.

A specific example of such a case will be described with reference toFIGS. 12A and 12B. FIGS. 12A and 12B are (first and second) diagramsillustrating a specific example of a determination method in a casewhere the previous change amount one and two versions ago is included.Note that FIG. 12A illustrates a progress of the change amount of theedge intensity in the case of the attachable matter, and FIG. 12Billustrates a progress of the change amount of the edge intensity in thecase of the white line.

In any of FIGS. 12A and 12B, it is difficult to see a significantdifference in the progress between the current change amount and theprevious change amount one version ago. In this case, it is possible toimprove the attachable matter detection accuracy by determining theprogress additionally including the previous change amount two versionsago.

Specifically, referring to FIG. 12A, in the case of an attachablematter, it is recognized that a progress between the previous changeamount two versions ago and the previous change amount one version agoindicated in a portion surrounded by a closed curve M1 of a dotted linehas no significant change. In addition, in FIG. 12A, it is recognizedthat the change amount abruptly decreases in the progress between theprevious change amount one version ago and the current change amountafter the aforementioned progress.

This indicates a characteristic of the raindrop that unexpectedlyadheres to blur the camera image and reduces the edge intensity of the“high.” Therefore, in a case where the progress of the change amount ofFIG. 12A is exhibited, it is possible to determine that this is anattachable matter.

Meanwhile, referring to FIG. 12B, in the case of a white line, it isrecognized that the progress of the change amount between the previouschange amount two versions ago and the previous change amount oneversion ago indicated by a closed curve M2 of a dotted line graduallydecreases from the “+” side. This progress is different from thecharacteristic of the raindrop illustrated in FIG. 12A. Therefore, in acase where the progress of the change amount of FIG. 12B is exhibited,it is possible to determine that this is not an attachable matter.

In this manner, by determining the progress of the change amountincluding previous change amounts, it is possible to improve theattachable matter detection accuracy. In addition, since it isdetermined whether or not an attachable matter is adhered on the basisof the temporal progress of the change amount of the histogram includingthe edge intensity, it is possible to detect an attachable matter withhigh accuracy, for example, even when a water repellent coating of thelens 2 a is degraded, and a blur raindrop having an unclear contour isattached.

Returning to FIG. 3, in the attachable matter detection apparatus 10according to the second embodiment, the condition adjustment unit 11 dadjusts the detection condition and performs a process of appropriatelyupdating the condition information 12 a when a predetermined triggersuitable for adjustment of the detection condition is generated.

A specific example of this case will be described with reference to FIG.13. FIG. 13 is a diagram illustrating a specific example of adjustmentof the detection condition. As illustrated in FIG. 13, as a trigger foradjusting the detection condition, for example, a “partitioned areaposition” may be employed.

In this case, as an example of the adjustment content, the change amounttends to be steep in the “lower” side of the screen as described above.Therefore, for example, if the “partitioned area position” is placed inthe “lower” side of the screen, the condition adjustment unit 11 dperforms adjustment for reinforcing the condition to prevent thedetection sensitivity from being excessively high accordingly.

As another trigger for adjusting the detection condition, for example, a“travel state,” may be employed. In this case, for example, while thevehicle C stops, an attachable matter such as raindrops is easilyattached compared to a traveling state. Therefore, in this case, forexample, the condition adjustment unit 11 d performs adjustment forloosening the condition to increase the detection sensitivity as anexample of the adjustment content.

As further another trigger for adjusting the detection condition, forexample, “wiper operation,” “rain sensor,” “rainfall informationreception,” or the like may be employed. All of them indicate asituation in which an attachable matter is easily attached due to arainfall or the like. Therefore, in this case, for example, thecondition adjustment unit 11 d performs adjustment for loosening thecondition to increase the detection sensitivity as an example ofadjustment content.

As still another trigger for adjusting the detection condition, forexample, a “color of the sky” may be employed. If it is detected thatthe sky is cloudy or rainy from the color of the sky through the imageanalysis of the camera image, similarly to the “wiper operation”described above, for example, the condition adjustment unit 11 dperforms adjustment for loosening the condition as an example of theadjustment content.

As still further another trigger for adjusting the exclusion condition,for example, a “gyro sensor” may be employed. For example, if it isdetected that the vehicle C is traveling downhill using the gyro sensor,similarly to the “wiper operation” described above, the conditionadjustment unit 11 d performs adjustment for loosening the condition toincrease the detection sensitivity.

Note that this is because, if the vehicle is traveling downhill, forexample, the lens 2 a of the rear camera 2-2 faces upward compared to anormal operation, so it can be said that raindrops are easily attachedin the case of rainfall or the like.

Next, a processing sequence executed by the attachable matter detectionapparatus 10 according to this embodiment will be described withreference to FIG. 14. FIG. 14 is a flowchart illustrating a processingsequence executed by the attachable matter detection apparatus 10according to the second embodiment.

First, the target area acquirement unit 11 a acquires each partitionedarea from a single frame of the camera image of the camera 2 (StepS201). In addition, the histogram creation unit 11 b creates each of theedge intensity histogram, the luminance histogram, and the saturationhistogram of each partitioned area acquired by the target areaacquirement unit 11 a (Step S202).

In addition, the attachable matter determination unit 11 c determineswhether or not there is an attachable matter for each partitioned areaon the basis of the change amount between frames of each histogramcreated by the histogram creation unit 11 b (Step S203).

Here, if it is determined that there is an attachable matter (Step S204,Yes), the attachable matter determination unit 11 c notifies the removaldetermination device 5, for example, of this partitioned area (StepS205). In addition, if it is determined that there is no attachablematter (Step S204, No), the attachable matter determination unit 11 cadvances the control to Step S206.

Then, the control unit 11 determines whether or not there is aprocessing end event (Step S206). The processing end event correspondsto, for example, IG OFF, ACC OFF, and the like. Here, if it isdetermined that there is no processing end event (Step S206, No), theprocess is repeated from Step S201. Otherwise, if it is determined thatthere is a processing end event (Step S206, Yes), the attachable matterdetection apparatus 10 terminates the process.

As described above, in the attachable matter detection apparatus 10according to the second embodiment, the attachable matter determinationunit 11 c determines whether or not there is an attachable matter on thebasis of the change amount between the current frame and the previousframe in the histogram created by the histogram creation unit 11 b.

Therefore, using the attachable matter detection apparatus 10 accordingto the second embodiment, it is possible to easily detect an attachablematter on the basis of the characteristic of raindrops exhibited in thechange amount between frames even in the case of, for example, raindropshaving a blur contour so that it is difficult to detect an edge. Thatis, it is possible to improve the attachable matter detection accuracy.

Third Embodiment

Next, a configuration of the removal control device 50 that has afunction of the attachable matter detection apparatus 10 described aboveand controls the whole processes including detection of an attachablematter to removal of the attachable matter according to a thirdembodiment will be described with reference to FIGS. 15 to 19.

FIG. 15 is a block diagram illustrating an attachable matter removalsystem 1 according to a third embodiment. Note that, in FIG. 15, onlyelements necessary to describe features of this embodiment areillustrated as functional blocks, and general elements are notillustrated intentionally.

In other words, each element illustrated in FIG. 15 is functional andconceptual, and is not necessarily configured as illustrated in aphysical sense. For example, a distributed and/or integrated version ofeach functional block is not limited to those illustrated, and itsentirety or a part thereof may be functionally or physically distributedor integrated in an arbitrary unit depending on various loads, usesituations, and the like.

FIG. 16A is a diagram illustrating an exemplary content of notificationfrom an attachable matter detector 51 a. In addition, FIG. 16B is adiagram illustrating an exemplary content of data regarding thedetection area Da contained in a detection information database (DB) 52a. Furthermore, FIG. 16C is an explanatory diagram illustrating a stateof the detection area Da.

As illustrated in FIG. 15, the attachable matter removal system 1includes a camera 2, an attachable matter remover 3, and a removalcontrol device 50. Since the camera 2 and the attachable matter remover3 have been described above, they will not be described here.

The removal control device 50 includes a control unit 51 and a memoryunit 52. The control unit 51 includes a plurality of attachable matterdetectors 51 a (for example, attachable matter detectors 51 a-1, 51 a-2,51 a-3, . . . ), an exclusion unit 51 b, an inter-algorithm overlapdetermination unit 51 c, an inter-frame overlap determination unit 51 d,an attachment determination unit 51 e, a removal necessity determinationunit 51 f, and an instruction unit 51 g.

The memory unit 52 is a memory device such as a hard disk drive, anonvolatile memory, and a register and stores the detection informationDB 52 a.

The control unit 51 controls the entire removal control device 50. Eachof a plurality of attachable matter detectors 51 a acquires a cameraimage of a single frame from the camera 2 and extracts a detection areaDa estimated to have an attachable matter out of the camera image usinga corresponding detection algorithm. In addition, the attachable matterdetector 51 a notifies the exclusion unit 51 b of the extracteddetection area Da.

Note that the attachable matter detection apparatus 10 according to thesecond embodiment described above corresponds to any one of theplurality of attachable matter detectors 51 a according to thisembodiment described above.

Here, as illustrated in FIG. 16A, the content of the notification fromthe attachable matter detector 51 a includes, for example, an upper leftcoordinate (x,y), a width w, and a height h of the detection area Daextracted as a rectangular shape.

The exclusion unit 51 b performs an image analysis for each detectionarea Da notification of which is provided from the attachable matterdetector 51 a and determines whether or not such an attachable matterestimated to exist in the detection area Da is truly an attachablematter.

The exclusion unit 51 b notifies the inter-algorithm overlapdetermination unit 51 c of the detection area Da determined as having anattachable matter as a result of the determination. Otherwise, theexclusion unit 51 b does not notify the inter-algorithm overlapdetermination unit 51 c of a detection area Da determined as having noattachable matter as a result of the determination, but excludes it fromthe processing target of the rear stage. In this manner, by excluding anunnecessary image area, it is possible to improve the attachable matterdetection accuracy and reduce a processing load of the rear stage.

Note that the attachable matter detection apparatus 10 of the firstembodiment described above corresponds to the exclusion unit 51 baccording to this embodiment.

The inter-algorithm overlap determination unit 51 c determines anoverlap of the detection area Da between a plurality of algorithms inthe current frame, that is, whether or not there is an overlappingportion between the detection areas Da extracted from each of theattachable matter detectors 51 a. In addition, the inter-algorithmoverlap determination unit 51 c reflects the determination result as“score” of each detection area Da. The reflection result is managed onthe detection information DB 52 a. The determination process executed bythe inter-algorithm overlap determination unit 51 c will be describedbelow in more details with reference to FIGS. 17A to 17D.

The inter-frame overlap determination unit 51 d determines whether ornot there is any overlap with the detection area Da already extracted inthe previous frame for all processing results of the inter-algorithmoverlap determination unit 51 c of the current frame. In addition, theinter-frame overlap determination unit 51 d reflects the determinationresult in “score” and “state” of each detection area Da. The reflectionresult is managed on the detection information DB 52 a.

Here, as illustrated in FIG. 16B, the detection information DB 52 aincludes, for example, a “detection area ID” column, an “areainformation” column, a “score” column, and a “state” column. The“detection area ID” column stores an identifier of the detection areaDa, and the detection information DB 52 a is managed for each detectionarea ID described above.

The “area information” column stores the upper left coordinate (x,y),the width w, the height h, and the like of the detection area Daillustrated in FIG. 16A. The “score” column stores the current score ofeach detection areas Da. The “state” column stores the current state ofeach detection area Da.

As illustrated in the state machine diagram of FIG. 16C, each detectionarea Da can transfer to four states including “IDLE,” “HIDING,”“OBSERVATION,” AND “PENALTY.” The state “IDLE” refers to a“non-detection state,” that is, a state in which no attachable matter isattached. The state “HIDING” refers to a state in which “an attachablematter is likely to be attached.”

The state “OBSERVATION” refers to an “observation state after theremoval process” in which an attachable matter is removed by theattachable matter remover 3. The state “PENALTY” refers to a “state inwhich an attachable matter is continuously detected from thecorresponding area even after the removal process,” that is, a removalfailure or error detection state.

The inter-frame overlap determination unit 51 d updates the “score” ofeach detection area Da and transfers the “state” on the detectioninformation DB 52 a depending on the determination result.

Returning to FIG. 15, the attachment determination unit 51 e determines“attachment confirmation” of the attachable matter depending on the“STATE” and “SCORE” of the detection area Da of the detectioninformation DB 52 a.

The removal necessity determination unit 51 f determines whether or notan attachable matter removal operation is actually performed by theattachable matter remover 3 if the “attachment confirmation” isdetermined by the attachment determination unit 51 e. The processingperformed by the inter-frame overlap determination unit 51 d, theattachment determination unit 51 e, and the removal necessitydetermination unit 51 f will be described below in more details withreference to FIGS. 18A to 18E.

The instruction unit 51 g generates an instruction signal for allowingthe attachable matter remover 3 to perform a removal operation andtransmits this instruction signal to the attachable matter remover 3 ina case where the removal necessity determination unit 51 f determinesthat it is necessary to remove an attachable matter.

Note that the removal determination device 5 of the first and secondembodiments described above corresponds to the inter-algorithm overlapdetermination unit 51 c, the inter-frame overlap determination unit 51d, the attachment determination unit 51 e, the removal necessitydetermination unit 51 f, and the instruction unit 51 g according to thisembodiment.

Next, the determination process executed by the inter-algorithm overlapdetermination unit 51 c will be described in more details with referenceto FIGS. 17A to 17D. FIGS. 17A to 17D are (first to fourth) explanatorydiagrams illustrating the processing of the inter-algorithm overlapdetermination unit 51 c.

As described above, the inter-algorithm overlap determination unit 51 cdetermines whether or not there is an overlap of the detection area Dabetween a plurality of algorithms in the current frame as illustrated inFIG. 17A. Specifically, as illustrated in FIG. 17A, for example, it isdetermined whether or not there is an overlap between all of thedetection areas Da-1 of the attachable matter detection algorithm-1 andall of the detection areas Da-2 of the attachable matter detectionalgorithm-2. This determination is similarly performed between theattachable matter detection algorithm-1 and the attachable matterdetection algorithm-n or between the attachable matter detectionalgorithm-2 and the attachable matter detection algorithm-n.

Note that, as illustrated in FIG. 17B, for example, an overlap betweenthe detection area Da-1 and the detection area Da-2 is determined on thebasis of a distance d from the center.

For example, in a case where it is determined that there is an overlapbetween the detection area Da-1 and the detection area Da-2, theinter-algorithm overlap determination unit 51 c adds a point to thescores of the detection areas Da-1 and Da-2 as illustrated in FIG. 17C.

As a result, it is possible to express a fact that a possibility ofexistence of an attachable matter is higher in the detection areas Da-1and Da-2 having an overlap compared to the detection area Da having nooverlap.

For example, in a case where there is an overlap in the detection areasDa-1 and Da-2 as illustrated in FIG. 17D, the inter-algorithm overlapdetermination unit 51 c updates the area information of the detectionareas Da-1 and Da-2.

For example, as illustrated in (a) of FIG. 17D, the area information isintegrated in the detection area Da-1 by prioritizing the detection areaDa-1. In addition, as illustrated in (b) of FIG. 17D, the areainformation is integrated in the detection area Da-2 by prioritizing thedetection area Da-2 on the contrary.

As illustrated in (c) of FIG. 17D, a logical product is taken, and thearea information is integrated into the detection area Da-A for only theoverlapping portion. In addition, as illustrated in (d) of FIG. 17D, thearea information may be integrated into the detection area Da-Scorresponding to a logical sum of the detection areas Da-1 and Da-2.

As illustrated in (e) of FIG. 17D, the area information may beintegrated into the detection area Da-E expanded to include both thedetection area Da-1 and the Da-2.

Next, the processing executed by the inter-frame overlap determinationunit 51 d, the attachment determination unit 51 e, and the removalnecessity determination unit 51 f will be described in details withreference to FIGS. 18A to 18E. FIGS. 18A to 18C are (first to third)explanatory diagrams illustrating the processing of the inter-frameoverlap determination unit 51 d.

FIG. 18D is an explanatory diagram illustrating the processing of theinter-frame overlap determination unit 51 d and the attachmentdetermination unit 51 e. In addition, FIG. 18E is an explanatory diagramillustrating the processing of the removal necessity determination unit51 f.

As described above, the inter-frame overlap determination unit 51 ddetermines whether or not there is an overlap with each detection areaDa where extraction has been completed for the previous frame for all ofthe processing results of the inter-algorithm overlap determination unit51 c regarding the current frame as illustrated in FIG. 18A.

Specifically, as illustrated in FIG. 18B, it is determined whether ornot there is an overlap between all of the detection areas Da-C of thecurrent frame and all of the detection areas Da-P of the previous frame.Note that a solution of the overlap may be similar to that of theinter-algorithm overlap determination unit 51 c.

As illustrated in FIG. 18B, if it is determined that there is an overlapbetween the detection area Da-C of the current frame and the detectionarea Da-P of the previous frame, the inter-frame overlap determinationunit 51 d adds a point to each score of the detection areas Da-C andDa-P.

As a result, the inter-frame overlap determination unit 51 d canindicate an attachable matter existing in substantially the same area inthe lens 2 a, for example, between distant frames.

Meanwhile, in a case where there is no overlap between the detectionarea Da-P of the previous frame and the detection area Da-C of thecurrent frame as illustrated in FIG. 18C, the inter-frame overlapdetermination unit 51 d reduces the score.

The inter-frame overlap determination unit 51 d determines that thedetection area Da-C of the current frame having no overlap with any oneof the detection area Da-P of the previous frame is a new detection areaDa and newly registers it on the detection information DB 52 a.

As illustrated in FIG. 18D, the newly registered detection area Da isgiven a predetermined score and has a “HIDING” state. In addition, theinter-frame overlap determination unit 51 d and the attachmentdetermination unit 51 e transfer the state of the detection area Dadepending on the score of the detection area Da that changes through the“adding point” or “reducing point” described above from such a “HIDING”state.

For example, in a case where the score of the detection area Da havingthe “HIDING” state becomes equal to or lower than a predetermined pointas illustrated in FIG. 18D, the inter-frame overlap determination unit51 d transfers the state of this detection area Da from “HIDING” to“IDLE” (Step S11). As a result, it is possible to prevent an erroneousresponse in which a removal process is performed, for example, for anattachable matter such as raindrops not necessary to remove because theyflow down, move, and the like.

In a case where the score of the detection area Da having the “HIDING”state becomes equal to or higher than a predetermined point, theattachment determination unit 51 e confirms attachment of the attachablematter for this area (attachment confirmation) (Step S12).

After the attachment confirmation, the attachment determination unit 51e transfers all the detection area Da having the “HIDING” state to the“OBSERVATION” state (Step S13). This is because, in a case where aremoval process is performed in response to attachment confirmation of asingle detection area Da, it is estimated that an attachable matter isremoved from other detection areas Da having the “HIDING” state that arenot determined as attachment confirmation in a normal case.

Note that, in a case where the score of the detection area Da having the“OBSERVATION” state through the removal process becomes equal to orhigher than a predetermined point, the inter-frame overlap determinationunit 51 d transfers the detection area Da to the “PENALTY” state (StepS14). As a result, it is possible to recognize a removal failure orerror detection in which an attachable matter is continuously detectedeven after the removal process.

In a case where the score of the detection area Da having the“OBSERVATION” state or the “PENALTY” state becomes equal to lower than apredetermined point, the inter-frame overlap determination unit 51 dtransfers the detection area Da to the “IDLE” state (Step S15).

Note that, in FIG. 18D, a response speed until attachment confirmationmay be controlled by adjusting a slope of the arrow indicating “addingpoint” or “reducing point.” For example, it is possible to increase theresponse speed taken from detection of an attachable matter to theremoval process by steeply changing the slope of the arrow by increasingthe added point and the reduced point.

The removal process may not be performed even for a detection area Dasubjected to the attachment confirmation. For example, as illustrated inFIG. 18E, the removal necessity determination unit 51 f determines thatexecution of the removal process is not necessary if the detection areaDa subjected to the attachment confirmation exists in a skip areasubstantially extending along the outer circumference of the screen.

In this manner, it is possible to reduce a processing load of the entiresystem by skipping the removal process for an attachable matter attachedto an image area less influencing visibility of a passenger or a drivingoperation.

Next, a processing sequence executed by the attachable matter removalsystem 1 according to this embodiment will be described with referenceto FIG. 19. FIG. 19 is a flowchart illustrating a processing sequenceexecuted by the attachable matter removal system 1 according to thethird embodiment.

First, each of a plurality of attachable matter detectors 51 a acquiresa camera image of a single frame (Step S301). In addition, for example,the attachable matter detector 51 a-1 extracts a detection area Da-1using an attachable matter detection algorithm-1 (Step S302).

For example, the attachable matter detector 51 a-2 extracts a detectionarea Da-2 using the attachable matter detection algorithm-2 (Step S303).In addition, for example, the attachable matter detector 51 a-n extractsa detection area Da-n using the attachable matter detection algorithm-n(Step S304).

The exclusion unit 51 b performs an exclusion process for each ofdetection areas Da extracted and notification of which is provided bythe attachable matter detector 51 a (Step S305). That is, the exclusionunit 51 b determines whether or not the attachable matter estimated toexist in the detection area Da is truly an attachable matter. If this isnot an attachable matter, the exclusion unit 51 b excludes thecorresponding detection area Da from the processing target of the rearstage.

Note that, for example, this exclusion process itself may be omitted. Asa result, it is possible to reduce a processing load of the entiresystem.

Subsequently, the inter-algorithm overlap determination unit 51 cperforms an inter-algorithm overlap determination process (Step S306).That is, the inter-algorithm overlap determination unit 51 c determineswhether or not there is an overlap of the detection area Da between aplurality of frames in the current frame and updates the score of thedetection area Da depending on the determination result.

The inter-frame overlap determination unit 51 d performs an inter-frameoverlap determination process (Step S307). That is, the inter-frameoverlap determination unit 51 d determines whether or not there is anoverlap for all of the processing results of the inter-algorithm overlapdetermination unit 51 c with each detection area Da from whichextraction has been completed in the previous frame and updates thescore and the state of the detection area Da depending on thedetermination result.

The attachment determination unit 51 e performs an attachmentdetermination process (Step S308). That is, the attachment determinationunit 51 e determines attachment confirmation of an attachable matterdepending on the score and the state of the detection area Da of thedetection information DB 52 a updated by the inter-frame overlapdetermination unit 51 d.

If the attachment determination unit 51 e determines “attachmentconfirmation,” the removal necessity determination unit 51 f determineswhether or not it is necessary to actually remove an attachable matterusing the attachable matter remover 3 (Step S309).

Here, if it is determined that it is necessary to remove an attachablematter (Step S309, Yes), the instruction unit 51 g outputs aninstruction signal to the attachable matter remover 3 to allow theattachable matter remover 3 to perform a removal process (Step S310).Meanwhile, if it is determined that it is not necessary to remove anattachable matter (Step S309, No), the instruction unit 51 g does notexecute the removal process.

The control unit 51 determines whether or not there is a processing endevent (Step S311). The processing end event corresponds to, for example,IG OFF, ACC OFF, and the like. Here, if it is determined that there isno processing end event (Step S311, No), the process is repeated fromStep S301. Otherwise, if it is determined that there is a processing endevent (Step S311, Yes), the attachable matter removal system 1terminates the processing.

According to the third embodiment described above, for example, a casewhere the attachable matter detection apparatus 10 of the secondembodiment executes one of the attachable matter detection algorithms-1,-2, . . . , n depending on any one of a plurality of attachable matterdetectors 51 a has been described by way of example. However, an exampleof the attachable matter detection algorithm is not limited to those ofthe second embodiment.

In this regard, in the following description, attachable matterdetection apparatuses 10A, 10B, and 10C according to the fourth to sixthembodiments for executing individual attachable matter detectionalgorithms will be described. Note that, in the following description, acase where a water droplet as the attachable matter is attached to thelens 2 a will be described by way of example.

First, before describing the fourth to sixth embodiments, an overview ofthe attachable matter detection method according to the fourth to sixthembodiments will be described with reference to FIG. 20. FIG. 20 is adiagram illustrating an overview of the attachable matter detectionmethod according to the fourth to sixth embodiments.

As illustrated in FIG. 20, in the attachable matter detection methodaccording to the fourth to sixth embodiments, first, edge information isextracted from the camera image L (Step S21). Here, the edge informationrefers to, for example, a gradient of the luminance in the horizontal(X-axis) and vertical (Y-axis) directions of the drawings in each pixelof the camera image L.

Then, in the attachable matter detection method according to the fourthto sixth embodiments, each pixel of the camera image L is converted intoa predetermined data format on the basis of this edge information (StepS22). Here, in the attachable matter detection method according to thefourth to sixth embodiments, by converting each pixel into apredetermined data format on the basis of the edge information, it ispossible to improve detection accuracy of water droplets.

Specifically, each pixel is binarized on the basis of the edge intensityof each pixel in the camera image L. As a result, it is possible toprevent influence from luminance unevenness of the water dropletappearing in the camera image L. That is, it is possible to accuratelydetect a water droplet that reflects light. This will be described indetails as a fourth embodiment with reference to FIGS. 21 to 27.

In the attachable matter detection method according to the fourth tosixth embodiment, each pixel is converted into a predetermined dataformat by using parameters whose edge directions opposite to the edgedirection of each pixel in the camera image L have a 1's complementrelationship.

As a result, it is possible to make clear a difference of the edgedirection of each pixel. For this reason, it is possible to improverecognition accuracy in the matching process. This will be described indetails as a fifth embodiment with reference to FIGS. 28 to 33.

In the attachable matter detection method according to the fourth tosixth embodiments, each pixel is encoded by allocating correspondingcodes to the edge directions of each pixel in the camera image L.

In this case, a matching process using a normalized expression with acode string indicating water droplets is performed. As a result, in thecamera image L, for example, it is possible to extract, for example,code strings of each side of the rectangle contained in the waterdroplet.

By detecting a water droplet attached to the camera 2 by combining theextracted code strings, it is possible to detect an irregular shapewater droplet such as a water droplet that has been cut off from thecamera image L. This will be described in details as a sixth embodimentwith reference to FIGS. 34 to 39.

In the attachable matter detection method according to the fourth tosixth embodiments, a matching process is performed between eachconverted pixel and a template representing a water droplet (Step S23),and on the basis of the matching result, a water droplet attached to thecamera 2 is detected (Step S24). Note that, in the camera image L1illustrated in the same drawing, a mark M is indicated in a portionwhere a water droplet is detected by the attachable matter detectionmethod.

In this manner, in the attachable matter detection method according tothe fourth to sixth embodiments, each pixel of the camera image L isconverted into a predetermined data format on the basis of edgeinformation, and a water droplet attached to the camera 2 is detectedthrough a matching process using such a data format.

Therefore, using the attachable matter detection method according to thefourth to sixth embodiments, it is possible to improve detectionaccuracy of a water droplet.

Note that, in the attachable matter detection method according to thefourth to sixth embodiments, a template indicating partial shapes ofdifferent parts of a water droplet is used depending on a scanningposition of the matching process. This will be described below in moredetails with reference to FIGS. 23 and 24.

Fourth Embodiment

Next, a configuration of an attachable matter detection apparatus 10Aaccording to a fourth embodiment will be described. FIG. 21 is a blockdiagram illustrating an attachable matter detection apparatus 10Aaccording to the fourth embodiment. Note that, in FIG. 21, the camera 2and the attachable matter remover 3 described above are alsoillustrated. Note that the attachable matter detection apparatus 10A mayalso be provided in each camera 2.

Although, in the aforementioned example, the attachable matter remover 3injects the compressed air and the washing liquid toward the lens 2 a ofthe camera 2 (refer to FIG. 1B), the present application is not limitedthereto. Instead, at least any one of the compressed air and the washingliquid may be injected. In addition, the attachable matter remover 3 maybe configured to wipe out the lens 2 a with a wiper.

The attachable matter detection apparatus 10A according to the fourthembodiment has a control unit 20A and a memory unit 30A. The controlunit 20A includes an image acquirement unit 21, an extraction unit 22, aconversion unit 23, a matching unit 24, and a detection unit 25. Inaddition, the memory unit 30A stores binarization threshold valueinformation 31, template information 32, and detection information 33.

The control unit 20A includes, for example, a central processing unit(CPU), a read-only memory (ROM), a random access memory (RAM), a harddisk drive (HDD), a computer having input/output ports, or variouscircuits.

The CPU of the computer serves as the image acquirement unit 21, theextraction unit 22, the conversion unit 23, the matching unit 24, andthe detection unit 25 of the control unit 20A, for example, by readingand executing a program stored in the ROM.

At least any one or all of the image acquirement unit 21, the extractionunit 22, the conversion unit 23, the matching unit 24, and the detectionunit 25 of the control unit 20A may be configured of hardware such as anapplication specific integrated circuit (ASIC) or a field programmablegate array (FPGA).

The memory unit 30A corresponds to, for example, RAM or HDD. The RAM orHDD may store the binarization threshold value information 31, thetemplate information 32, the detection information 33, or information onvarious programs.

Note that the attachable matter detection apparatus 10A may acquire theprograms or information described above via computers connected via awired or wireless network or a portable recording medium.

The image acquirement unit 21 acquires a camera image from the camera 2and converts the camera image into a grayscale image through grayscaleconversion. In addition, the image acquirement unit 21 outputs thegrayscale image to the extraction unit 22.

Note that the grayscale conversion refers to a process of expressingeach pixel of the camera image in each gray scale from white to blackdepending on luminance. Note that the grayscale conversion process mayalso be omitted.

The extraction unit 22 applies a Sobel filter to the grayscale imageinput from the image acquirement unit 21 to extract edge information ofeach pixel in the grayscale image. Here, the edge information refers tothe edge intensity in the X-axis and Y-axis directions of each pixel.

The extraction unit 22 outputs the extracted edge information to theconversion unit 23 in associating with the grayscale image. Note thatthe extraction unit 22 may use another edge extraction method such as aLaplacian filter in place of the Sobel filter.

The conversion unit 23 binarizes the grayscale image on the basis of theedge information of each pixel input from the extraction unit 22.Specifically, first, the conversion unit 23 calculates a value obtainedby squaring each of the edge intensities of the X-axis and Y-axisdirections as the edge information and adding them as an edge amount ofeach pixel.

Subsequently, the conversion unit 23 binarizes the grayscale image bysetting a pixel having a calculated edge amount equal to or larger thana binarization threshold value THa described below to “1,” and setting apixel having a calculated edge amount equal to or smaller than thebinarization threshold value THa to “0.”

In this manner, the conversion unit 23 can cancel influence of noise bybinarizing each pixel on the basis of the edge amount. For this reason,it is possible to improve detection accuracy of water droplets. Inaddition, through the binarization, all edges of a water droplet are setto be equal. For this reason, it is possible to suppress a processingload caused by the matching unit 24.

Note that, although binarization is performed on the basis of the edgeamount in the following description, there is a matching relationshipbetween the edge intensity and the edge amount. For this reason,binarization based on the edge amount has the same meaning asbinarization based on the edge intensity.

The conversion unit 23 outputs the image obtained by binarizing eachpixel (hereinafter, referred to as a binarization image) to the matchingunit 24. Here, in the attachable matter detection apparatus 10Aaccording to the fourth embodiment, the binarization threshold value THais set dynamically depending on the surrounding environment of thecamera 2. This will be described below in more details with reference toFIG. 22.

Note that the conversion unit 23 may binarize the grayscale image on thebasis of characters such as “1” or “0” or other letters or symbols inaddition to “black” and “white.”

The matching unit 24 calculates similarity between the binarizationimage and the template through a matching process between thebinarization image input from the conversion unit 23 and the templaterepresenting characteristics of a water droplet. In addition, thematching unit 24 outputs the calculated similarity value to thedetection unit 25 in association with each pixel of the binarizationimage.

Note that the process of the matching unit 24 will be described below inmore details with reference to FIG. 25. In addition, the template isstored in the memory unit 30A as template information 32. The templatewill be described below in more details with reference to FIG. 23.

The detection unit 25 detects a water droplet attached to the camera 2on the basis of the similarity input from the matching unit 24. Inaddition, if a water droplet is detected, the detection unit 25 notifiesthe attachable matter remover 3 or a vehicle control apparatus (notillustrated) that performs automatic driving of the vehicle C of thefact.

As a result, for example, the attachable matter remover 3 removes thewater droplet attached to the camera 2. In addition, the vehicle controlapparatus, for example, recognizes a white lane by avoiding such anattached area. Note that the detection process of the detection unit 25will be described below in more details with reference to FIGS. 26A to26D.

Next, a method of setting the binarization threshold value THa using theconversion unit 23 will be described with reference to FIG. 22. FIG. 22is a diagram illustrating a binarization threshold value THa. Note thatthe ordinate of FIG. 22 refers to the edge amount described above, andthe abscissa refers to the surrounding illumination.

As illustrated in FIG. 22, the binarization threshold value THa is setdynamically, for example, depending on the illumination as thesurrounding environment. Specifically, the binarization threshold valueTHa is set to be higher, for example, as the surrounding illuminationincreases by setting the edge amount Sa as an upper limit, and settingthe edge amount Sb as a lower limit.

If a pixel has an edge amount larger than the binarization thresholdvalue THa, the pixel is set as “white” in the binarization image. If apixel has an edge amount equal to or smaller than the binarizationthreshold value THa, the pixel is set as “black” in the binarizationimage.

Here, light from a surrounding light source may be reflected on a waterdroplet. In this case, an edge amount of the pixel representing the edgeof the water droplet increases. Here, if the surrounding illumination ishigh, that is, if the surroundings are bright, the grayscale imageincludes background structures, white lanes on the road, and the like inaddition to the water droplet.

For this reason, by setting the binarization threshold value THa to ahigh value, only pixels indicating the edge of the water droplet are setto “white.” In other words, edges of an unnecessary object areeffectively removed by setting the binarization threshold value THa to ahigh value.

Meanwhile, if the illumination is low, that is, if the surroundings aredark, an unnecessary object is difficult to appear in the grayscaleimage. In addition, if light is reflected on a water droplet, the edgeamount of the pixel indicating the edge of the water droplet easilyincreases compared to the pixel indicating the edge of the unnecessaryobject.

Depending on the intensity of the light source, the edge amount of thewater droplet may not exceed the edge amount Sa in some cases. In thiscase, if the binarization threshold value THa in the case of darksurroundings is set to be equal to the binarization threshold value THain the case of a high illumination, the edge amount of the water dropletmay become lower than the binarization threshold value THa, which is notdesirable.

From this fact, when the surroundings are dark, by setting thebinarization threshold value THa to be lower than that of a case wherethe illumination is high, only the pixel indicating the edge of thewater droplet where light is reflected is set to “white.” In otherwords, in a dark place, it is possible to effectively extract the edgeof the water droplet regardless of the intensity of the light source.

In this manner, in the attachable matter detection apparatus 10Aaccording to the fourth embodiment, by setting the binarizationthreshold value THa depending on the surrounding situation, it ispossible to effectively extract only the edge of the water droplet.

As a result, it is possible to accurately detect a water droplet thatreflects light in the nighttime during which detection was difficult inthe related art or detect a water droplet when there is a strong lightsource in the daytime. Note that, as illustrated in FIG. 22, theconversion unit 23 sets the binarization threshold value THa, forexample, to the edge amount Sa and the edge amount Sb in the daytime andthe nighttime, respectively.

As a method of discriminating between daytime and nighttime, forexample, a method of discriminating between daytime and nighttimedepending on a time zone or a method of discriminating between daytimeand nighttime in conjunction with a headlight of the vehicle C may beemployed.

For example, when the headlight is ON, the conversion unit 23 may setthe binarization threshold value THa to the edge amount Sb. In addition,when the headlight is OFF, the conversion unit 23 may set thebinarization threshold value THa to the edge amount Sa.

For example, when the vehicle C has an illumination sensor, theconversion unit 23 may set the binarization threshold value THa on thebasis of a sensor value of the illumination sensor.

In this case, for example, the conversion unit 23 may sequentially setthe binarization threshold value THa depending on the illumination asindicated by the dotted line in FIG. 22. In addition, the binarizationthreshold value THa may be changed in response to a user manipulation ona manipulation unit (not illustrated).

Although a case where the binarization threshold value THa is changeddepending on the illumination as the surrounding environment here, thepresent application is not limited thereto. That is, the binarizationthreshold value THa may be changed on the basis of positionalinformation as the surrounding environment.

For example, the conversion unit 23 may acquire positional informationfrom a navigation device and set the binarization threshold value THa toa high value, for example, in a case where the vehicle C is placedindoors, such as a multi-grade parking lot.

Next, a template according to the fourth embodiment will be describedwith reference to FIG. 23. FIG. 23 is a diagram illustrating anexemplary template according to the fourth embodiment. As illustrated inFIG. 23, in the attachable matter detection apparatus 10A according tothe fourth embodiment, a template having a white contour of a waterdroplet on a black background is employed.

As illustrated in FIG. 23, the attachable matter detection apparatus 10Aaccording to the fourth embodiment has a plurality of templatesindicating characteristics of the contour of the water droplet.Specifically, the template has a contour of a water droplet having asubstantially perfect circular shape as illustrated in (a) of FIG. 23 orhas a partial shape of the water droplet having a substantially semi-arcshape as illustrated in (b) to (e) of FIG. 23.

As illustrated in (f) of FIG. 23, the template may have a substantiallyelliptical shape without limiting to a substantially perfect circularshape or a substantially semicircular shape. This is because, if aconvex lens such as a fisheye lens is employed in the camera 2, waterdroplets attached to the convex lens tend to have an elliptical shape.

In this manner, in the attachable matter detection apparatus 10Aaccording to the fourth embodiment, it is possible to improve detectionaccuracy of water droplets by performing a matching process using aplurality of templates. In addition, by using the template havingpartial shapes of water droplets as illustrated in (b) to (e) of FIG.23, it is possible to detect a water droplet even when this waterdroplet can be removed from the image.

In the attachable matter detection apparatus 10A according to the fourthembodiment, an area for performing the matching process is set for eachtype of the template (refer to (a) to (f) of FIG. 23). This will bedescribed below in more details with reference to FIG. 24.

Note that the template of FIG. 23 is just for exemplary purposes and isnot limited to that illustrated. For example, an optimum template may beobtained on the basis of simulation or statistics in consideration of ashape of the camera 2, dripping of water droplets, and the like.

The attachable matter detection apparatus 10A according to the fourthembodiment has a plurality of templates having different scales asillustrated in (a) to (f) of FIG. 23. As a result, it is possible toaccurately detect water droplets having different sizes. Note that thematching unit 24 may perform a matching process from templates havingdifferent sizes depending on the surrounding environment of the camera2.

For example, in heavy rain, since large rain drops are easily attachedto the camera 2, the matching unit 24 starts the matching process from alarge template for large water droplets and performs the matchingprocess by gradually reducing the size of the template.

In light rain, since small rain drops are easily attached to the camera2, the matching unit 24 starts the matching process from a smalltemplate and performs the matching process by gradually enlarging thesize of the template.

As a result, it is possible to effectively detect a water dropletdepending on a surrounding situation of the vehicle C. In addition, inthe attachable matter detection apparatus 10A according to the fourthembodiment, different detection threshold values are set depending onthe size of the template. This will be described below in more detailswith reference to FIG. 26A.

Next, a relationship between the template and the scanning position willbe described with reference to FIG. 24. FIG. 24 is a diagramillustrating a relationship between the template and the scanningposition. It is assumed that the regions Ra to Re of FIG. 24 correspondto the templates of (a) to (e) of FIG. 23, respectively.

Specifically, the matching unit 24 is used when the region Ra positionedin the center of the binarization image L2 is scanned using the templatehaving a substantially perfect circular shape illustrated in (a) of FIG.23. In addition, the template having an arc shape facing downward in (b)of FIG. 23 is used when the region Rb positioned in the upper side ofthe binarization image L2 is scanned.

The template having an arc shape facing upward in (c) of FIG. 23 is usedwhen the region Rc positioned in the lower side of the binarizationimage L2 is scanned. Similarly, the templates having arc shapes facingleftward and rightward in (d) and (e) of FIG. 23 are used when theregions Rd and Re positioned in the left and right sides of thebinarization image L2 are scanned.

In this manner, the matching unit 24 may perform the matching processusing different templates depending on the scanning position of thebinarization image L2. As a result, it is possible to effectively detecta water droplet having a shape that can be easily attached to eachregion of the binarization image L2.

Note that the matching unit 24 may use all types of templates to performscanning for the entire area of the binarization image L2. In this case,it is possible to suppress a failure to detect a water droplet.

In a case where the camera 2 has a wide-angle lens, a water droplet isdistorted as close to the edge of the camera image. For this reason, atemplate having a characteristic of a distorted water droplet contour asclose to the edge may be employed.

Next, a template and a matching process using the matching unit 24 willbe described with reference to FIG. 25. FIG. 25 is a diagramillustrating an exemplary matching process.

As illustrated in FIG. 25, first, the matching unit 24 places, forexample, the upper left side of the template G in the upper left pixelP1 of the binarization image L2.

Subsequently, the matching unit 24 calculates similarity between thebinarization image L2 and the template G in this position. Note that acalculation method of this similarity will be described in the fifthembodiment, and thus, will not be described here.

Subsequently, the matching unit 24 stores the calculated similarityvalue, for example, in the pixel P1 positioned in the upper left side ofthe template G. Then, the matching unit 24 shifts the template G to theright by a single pixel and calculates the similarity. The calculatedsimilarity value is stored in the pixel P2.

The matching unit 24 repeats calculation of the similarity in thismanner to the right end. If the calculation of similarity is completedto the right end, the calculation process is repeated by shifting thetemplate G downward by a single pixel, so that the similaritycalculation is performed for overall pixels. As a result, the matchingunit 24 obtains the similarity values for overall pixels.

Similarly, the matching unit 24 also calculates the similarity fortemplates having different types or sizes. In addition, the matchingunit 24 outputs the calculated similarity values to the detection unit25 in association with coordinates of the pixels.

Here, the matching unit 24 does not necessarily calculate the similarityvalues for overall pixels. For example, the similarity calculationprocess may be simplified by calculating the similarity at apredetermined interval. As a result, it is possible to reduce aprocessing load caused by the matching unit 24.

For example, as in the region Rg positioned in the center of thebinarization image L2 of FIG. 25, the similarity value may be calculatedfrom a region having a higher priority in water droplet detection. As aresult, it is possible to quickly detect attached water droplets in ahigh priority region.

In this case, the matching unit 24 may not calculate the similarity fora region having a low priority such as an upper area of the binarizationimage L2. As a result, it is possible to further reduce the processingload caused by the matching unit 24.

For example, the matching unit 24 may calculate similarity for overallpixels of a region having a high priority, and may simplify thesimilarity calculation process for regions having a low priority.

That is, water droplets may be detected with high accuracy in a highpriority region and may be detected with rough accuracy in a lowpriority region. As a result, it is possible to suppress a failure todetect a water droplet in the high priority region while suppressing theprocessing load.

Next, a determination process using the detection unit 25 will bedescribed with reference to FIGS. 26A to 26D. FIGS. 26A to 26D are(first to fourth) diagrams illustrating a determination process usingthe detection unit 25.

First, a relationship between the template and the detection thresholdvalue will be described with reference to FIG. 26A. As illustrated inFIG. 26A, the detection unit 25 prepares different detection thresholdvalues depending on the size of the template.

Note that the detection threshold value is used by the detection unit 25to compare with the similarity value input from the matching unit 24. Inaddition, the detection unit 25 determines that a water droplet isattached if the similarity value is equal to or higher than thedetection threshold value.

As illustrated in FIG. 26A, in the attachable matter detection apparatus10A according to the fourth embodiment, the detection threshold value isset to be lower as the template is larger. Meanwhile, the detectionthreshold value is set to higher as the template is smaller.

This is because an unnecessary edge other than a water droplet can beeasily detected as an erroneous edge of the water droplet as thetemplate is smaller. That is, by setting the detection threshold valuedepending on the size of the template, it is possible to suppresserroneous detection of a water droplet.

Note that, although FIG. 26A illustrates a case where the template has asubstantially perfect circular shape corresponding to (a) of FIG. 23,this similarly applies to other templates illustrated in (b) to (f) ofFIG. 23. In addition, different detection threshold values may beprovided depending on the type of the template. Alternatively, the samedetection threshold value may be employed for overall templates.

Next, a case where the detection unit 25 provides different detectionthreshold values for each region of the binarization image L2 will bedescribed with reference to FIG. 26B. As described above, a priority fordetecting a water droplet is differently set depending on a region ofthe binarization image L2.

In this regard, in the attachable matter detection apparatus 10Aaccording to the fourth embodiment, different detection threshold valuesmay be provided depending on the region of the binarization image L2.Specifically, for example, for a region R1 where a position close to thevehicle C appears, the water droplet detection priority is set tohigher, and the detection threshold value is set to be lower, compare toother regions. As a result, it is possible to suppress a failure todetect a water droplet in the high priority region R1.

Meanwhile, for a region Rh where a position far from the vehicle Cappears, the water droplet detection priority is set to be low, and thedetection threshold value is set to be high. In addition, for a regionRm located between the regions R1 and Rh, for example, the detectionthreshold value is set to a median value between those of the regions R1and Rh.

In this manner, by setting different detection threshold valuesdepending on the region of the binarization image L2, it is possible tosecurely detect a water droplet in the high priority region R1 whilereducing erroneous detection of a water droplet in the low priorityregion Rh.

Note that, although a case where the binarization image L2 is dividedinto three regions has been described in FIG. 26B, the binarizationimage L2 may be divided into two regions, or four or more regions. Inaddition, the detection threshold value may be set by dividing thebinarization image L2 in a horizontal direction, in a slope direction,or in a concentric circular shape.

The priority of each region described above is just for exemplarypurposes, and the priority may be changed depending on a purpose ofwater droplet detection. In addition, the detection unit 25 may setdifferent detection threshold values, for example, between daytime andnighttime.

Next, a case where the detection unit 25 detects a water droplet on thebasis of the binarization images L2 of a plurality of frames will bedescribed with reference to FIG. 26C. Here, for example, a sceneappearing when the detection unit 25 completes the detection processdescribed above with reference FIGS. 26A and 26B is illustrated.

FIG. 26C illustrates a case where similarity values with the templatescorresponding to the sizes of each frame in the region where frames F1to F3 exist are higher than the detection threshold value.

Here, as in the frames F1 and F2, in a case where there is a regionwhere the similarity with a template having a different size is higher,and the frame F2 exists in a region surrounded by the frame F1, thedetection unit 25 determines that a water droplet is attached to such aregion. This is because, in a case where the similarity is high in asingle template, the similarity tends to increase in other templateshaving a similar size.

For example, since the aforementioned condition is not satisfied in acase where only a single frame exists in the neighboring region as inthe frame F3, the detection unit 25 determines that a water droplet isnot attached to the region of such a frame F3. As described above, in acase where the similarity is high in a single template, the similaritytends to increase in other templates having a similar size. Therefore,if the similarity is high in only a single template, and the similarityis not high in other templates (having a similar size), it can bedetermined that the similarity in a single template incidentallyincreases due to influence of noise or the like.

That is, the detection unit 25 determines that a water droplet isattached for the region where the similarities with a plurality oftemplates having different sizes are high. In this manner, by detectinga water droplet on the basis of the similarity with a plurality oftemplates, the attachable matter detection apparatus 10A can improvereliability of water droplet detection accuracy while reducing erroneousdetection of a water droplet.

Note that, in this case, for example, the detection unit 25 maydetermine that a water droplet is attached when the frames F1 and F2 areoverlapped at least partially. In addition, the detection unit 25 maydetermine that a water droplet is attached, for example, when aplurality of frames exist in a close position.

Next, a case where the detection unit 25 detects a water droplet on thebasis of a plurality binarization images L2 will be described withreference to FIG. 26D. In FIG. 26D, it is assumed that the newerbinarization image L2 is arranged in the closer side on the paper plane.That is, the binarization images L2 are newer in order from thebinarization image L2 e to the binarization image L2 a.

Similarly to FIG. 26C, in the frames F4 and F5 illustrated in FIG. 26D,it is assumed that the similarity with the template corresponding to thesize of the frame is high in the region of these frames.

As illustrated in FIG. 26D, the frame F4 exists in the binarizationimage L2 a, the binarization image L2 b, and the binarization image L2c, and the frame F5 exists in the binarization image L2 a, thebinarization image L2 c, and the binarization image L2 e.

In this case, for example, in a case where the frame exists in the sameposition across three continuous frames, the detection unit 25determines that a water droplet is attached to such a region. In theexample of FIG. 26D, since the frame F4 is detected continuously acrossthree frames, the detection unit 25 determines that a water droplet isattached to the region of the frame F4.

Note that the detection unit 25 may determine that a water droplet isattached to the region of the frame F5, for example, if a frame isdetected three times across five frames as in the frame F5.

In this manner, since the detection unit 25 detects attachment of awater droplet using a plurality of frames, it is possible to improvedetection accuracy while suppressing erroneous detection of a waterdroplet. Note that the position or size of the frame F4 or F5 does notnecessarily strictly match between frames, and a predetermined margin isallowed.

In FIG. 26D, a water droplet is detected in a case where the detectionunit 25 detects the same frame continuously across three frames.However, the present application is not limited thereto. A water dropletmay be detected in a case where the same frame is continuously detectedacross two or less frames, or four or more frames.

Note that the detection unit 25 may change the number of frames used inwater droplet detection depending on the priority of the regiondescribed above or a purpose of water droplet detection. For example, itmay be possible to set a large number of frames in a region wheredetection accuracy is prioritized, and reduce the number of frames in aregion where a detection speed is prioritized.

Next, a processing sequence executed by the attachable matter detectionapparatus 10A according to the fourth embodiment will be described withreference to FIG. 27. FIG. 27 is a flowchart illustrating a processingsequence executed by the attachable matter detection apparatus 10Aaccording to the fourth embodiment. Note that the following processingis repeatedly executed by the control unit 20A.

As illustrated in FIG. 27, first, the image acquirement unit 21 acquiresa camera image from the camera 2 and performs grayscale conversion forthis camera image (Step S401). Subsequently, the extraction unit 22extracts edge information of each pixel from the grayscale image (StepS402).

Then, as illustrated in FIG. 22, the conversion unit 23 sets thebinarization threshold value THa (Step S403) and binarizes the grayscaleimage (Step S404).

Subsequently, as illustrated in FIG. 25, the matching unit 24 performs amatching process using the binarization image binarized by theconversion unit 23 (Step S405). In addition, as illustrated in FIGS. 26Ato 26D, the detection unit 25 detects a water droplet attached to thecamera 2 (Step S406).

As described above, the attachable matter detection apparatus 10Aaccording to the fourth embodiment can detect a water droplet with highaccuracy in both nighttime and daytime by performing binarization on thebasis of the edge amount of each pixel.

Meanwhile, the conversion process of the conversion unit 23 describedabove is not limited to the binarization. In this regard, next, a casewhere each pixel is converted into parameters depending on the edgedirection will be described as a fifth embodiment.

Fifth Embodiment

First, a configuration of the attachable matter detection apparatus 10Baccording to the fifth embodiment will be described with reference toFIG. 28. FIG. 28 is a block diagram illustrating an attachable matterdetection apparatus 10B according to the fifth embodiment. Note that, inthe following description, like reference numerals denote like elementsas in the aforementioned embodiments, and they will not be describedrepeatedly.

As illustrated in FIG. 28, the attachable matter detection apparatus 10Baccording to the fifth embodiment includes a control unit 20B and amemory unit 30B. The control unit 20B has a conversion unit 23B, amatching unit 24B, and a detection unit 25B in place of the conversionunit 23, the matching unit 24, and the detection unit 25 illustrated inFIG. 21.

The memory unit 30B stores range information 34, parameter information31B, template information 32B, and detection information 33B. Here, theconversion unit 23B will be described first, and the image acquirementunit 21 and the extraction unit 22 will not be described.

The conversion unit 23B calculates edge directions of each pixel on thebasis of the edge information of each pixel input from the extractionunit 22 and allocates three-dimensional parameters to these edgedirections to parameterize each pixel. This will be described below inmore details with reference to FIGS. 30A and 30B.

Before calculating the edge direction, the conversion unit 23Bcalculates the aforementioned edge amount and performs filtering for thecalculated edge amount. This will be described below in more detailswith reference to FIG. 29.

The matching unit 24B performs a matching process between theparameterized grayscale image input from the conversion unit 23B and thetemplate representing characteristics of a water droplet. This matchingprocess has been described above in conjunction with FIG. 25 and willnot be described repeatedly here.

Here, the matching unit 24B calculates similarity on the basis of azero-mean normalized cross-correlation. In this zero-mean normalizedcross-correlation, the similarity takes a value from “−1” to “+1.”

Note that the matching unit 24B may calculate the similarity on thebasis of other calculation methods such as a sum of absolute difference(SAD) method or a sum of squared difference (SSD) method.

The detection unit 25B detects a water droplet attached to the camera 2on the basis of the similarity input from the matching unit 24B. Notethat the detection process of the detection unit 25B will be describedbelow in more details with reference to FIG. 33.

Next, a filtering process using the conversion unit 23B will bedescribed in details with reference to FIG. 29. FIG. 29 is a diagramillustrating an extraction range W. Note that the ordinate of FIG. 29refers to the edge amount. In addition, in FIG. 29, the extraction rangeW is hatched.

As illustrated in FIG. 29, in the attachable matter detection apparatus10B according to the fifth embodiment, for example, different extractionranges W are set between nighttime and daytime. Specifically, in thenighttime, a range from the edge amount Sc to the edge amount Se is setas an extraction range W1.

In daytime, a range from the edge amount Sd smaller than the edge amountSc to the edge amount Sf is set as an extraction range W2 in order toeffectively extract only edges of a water droplet regardless of thesurrounding brightness as described above.

By performing filtering as described above, it is possible to remove anunnecessary edge other than a water droplet. As a result, it is possibleto improve water droplet detection accuracy.

Note that an optimum value of the extraction range W1 or W2 may bederived using simulation or statistics. In addition, the conversion unit23B may set the extraction range W by feeding back the water dropletdetection result of the detection unit 25B.

For example, in a case where the detection unit 25B does not detect awater droplet, there is a possibility that the extraction range W maynot be set appropriately. For this reason, the conversion unit 23B mayextract each pixel again by changing the extraction range W. As aresult, it is possible to suppress a failure to detect a water droplet.Therefore, it is possible to improve water droplet detection accuracy.

Statistic values such as a distribution or an average of the edgeamounts in each pixel for each of nighttime and daytime are stored inthe memory unit 30B. In addition, the conversion unit 23B may set theextraction range W by discriminating between nighttime and daytime bycomparing the statistic values with the extracted edge amounts.

As a result, it is possible to appropriately set the extraction range Wdepending on the surrounding situation. Note that these statistic valuesmay be derived by the control unit 20B if the control unit 20B has alearning capability.

Next, a processing of the conversion unit 23B will be described withreference to FIGS. 30A and 30B. FIG. 30A is a diagram illustrating anedge vector. FIG. 30B is a diagram illustrating an example of theparameter information 31B.

Note that FIG. 30A illustrates the edge intensity in the X-axis andY-axis directions. As described above, the conversion unit 23Bcalculates a vector of each pixel on the basis of the edge intensitiesof the X-axis direction and the Y-axis direction input from theextraction unit 22.

Specifically, the vector of each pixel is calculated using atrigonometric function based on the edge intensities in the X axisdirection and the Y axis direction. Hereinafter, an angle θ between thevector calculated from FIG. 30A and the X-axis on the positive directionside will be referred to as an edge direction, and the length L of thevector will be referred to as an edge intensity of each pixel. Note thatthis edge intensity is used in the sixth embodiment.

Note that the conversion unit 23B may not necessarily calculate the edgedirection for overall pixels. Instead, the processing may be simplifiedby calculating the edge direction for each pixel at a predeterminedinterval in a low priority region.

Subsequently, the conversion unit 23B parameterizes the calculated edgedirection using the parameter information 31B of FIG. 30B. Here, a casewhere the conversion unit 23B parameterizes the edge direction using acolor vector of a twelve-color wheel will be described.

Here, the color vector is a vector defined depending on R, G, and Bcolor elements and has a three component parameters including R, G, andB.

Note that, in the twelve-color wheel, if red, green and blue colors areexpressed in decimal numbers, each parameter is expressed in threevalues including “0,” “128 or 127,” and “255.” Here, in the twelve-colorwheel, each value of R, G, and B having a complementary colorrelationship has a 1's complement relationship in a binary notation suchas a hexadecimal notation.

That is, in the conversion unit 23B according to the fifth embodiment,each pixel is converted using parameters whose each value of R, G, and Bbetween opposite angle ranges satisfies the 1's complement relationship.Note that these parameters do not necessarily strictly satisfy the 1'scomplement relationship, and it is assumed that a predetermined marginis allowed.

Specifically, the conversion unit 23B allocates parameters of R, G, andB of the angle range corresponding to the calculated edge direction tothe pixel. For example, in a case where the edge direction has an anglerange of 75° to 105°, an RGB parameter (0, 255, 255) corresponding tolight blue is allocated to the parameter of this edge direction.

In a case where the edge direction exists in an angle range of −75° to−105° opposite to this angle range, an RGB parameter (255, 0, 0)corresponding to red, which is a complementary color of light blue, isallocated.

The conversion unit 23B uses an RGB parameter (255, 255, 255)corresponding to white for the pixel whose edge amount is out of theextraction range W.

In this manner, in the attachable matter detection apparatus 10Baccording to the fifth embodiment, each pixel is converted usingparameters having a 1's complement relationship between the oppositeedge directions. As a result, it is possible to clearly distinguish adifference between opposite edge directions.

Using the color vector of the twelve-color wheel, it is possible toclearly distinguish the difference between angle ranges corresponding tothe edge direction. For this reason, it is possible to improverecognition accuracy of the matching unit 24B.

In the attachable matter detection apparatus 10B according to the fifthembodiment, the same parameter is allocated as long as the edgedirection is close regardless of the edge intensity. Therefore, it ispossible to detect even a water droplet having a blurred edge, that is,a water droplet having a weak edge intensity with accuracy equal to thatof a water droplet having a strong edge intensity.

As a result, in the attachable matter detection apparatus 10B accordingto the fifth embodiment, it is possible to detect even a water droplethaving a weak edge intensity, which has been difficult to detect in therelated art, that is, a water droplet blurred in the camera image withhigh accuracy.

The table used as a reference by the conversion unit 23B is not limitedto that illustrated in FIG. 30B. For example, the angle range may bedivided more minutely into twenty four colors instead of twelve colors.Alternatively, the angle range may be divided in a larger size such asnine colors.

Without limiting to the RGB parameter, other parameters in whichopposite edge directions satisfy the 1's complement relationship mayalso be employed. For example, a matrix may be employed as theparameter.

Next, a template according to the fifth embodiment will be describedwith reference to FIG. 31. FIG. 31 is a schematic diagram illustratingthe template according to the fifth embodiment. Note that, in FIG. 31,the template is simplified for convenient description purposes.

As illustrated in FIG. 31, in the template according to the fifthembodiment, a circle simulating a water droplet is divided into aplurality of fan-shaped regions. In addition, parameters correspondingto the aforementioned angle range set such that values of oppositeregions with respect to the center of the circle have a 1's complementrelationship are allocated to each fan-shaped region.

As illustrated in (a) of FIG. 31, an RGB parameter representing whiteand gray is allocated to regions other than the region representingcharacteristics of a water droplet. Specifically, an RGB parameter (255,255, 255) corresponding to white is allocated to the center of thetemplate, and an RGB parameter (128, 128, 128) corresponding to gray isallocated to an outer side of a water droplet.

Note that parameters of the center region and the outer region of thewater droplet are just exemplary, and a parameter corresponding toanother color may also be allocated.

As illustrated in (b) of FIG. 31, a template in which each parameter isallocated in a radial shape from the center across the entire region maybe employed. This template is effective to detect a water droplet thathas a thin thickness and appears blurred on the camera image.

As illustrated in (c) of FIG. 31, the white region in the center of thetemplate may be enlarged. In a case where such a template is employed,it is possible to effectively detect a water droplet having a brightedge. In addition, as illustrated in (d) of FIG. 31, the shaperepresenting the water droplet may have an elliptical shape.

In this manner, in the attachable matter detection apparatus 10Baccording to the fifth embodiment, for a grayscale image in whichopposite edge directions are converted to have parameters having the 1'scomplement relationship, a template representing a water droplet usingsuch a parameter is employed.

Therefore, a difference between edge directions in each pixel and eachtemplate becomes clear, and it is possible to improve recognitionaccuracy of the matching unit 24B.

As described above in conjunction with (b) to (e) of FIG. 23, theattachable matter detection apparatus 10B according to the fifthembodiment may use a semicircular template. In this case, as describedabove in conjunction with FIG. 24, it may be possible to perform amatching process that illustrates a water droplet having a partial shapecorresponding to the scanning position.

Note that, in the attachable matter detection apparatus 10B according tothe fifth embodiment, the matching process is performed using a templaterepresenting characteristics of a water droplet in a case where theluminance increases from the end to the center, that is, the edgedirection is directed from the end to the center.

However, without limiting thereto, a template representingcharacteristics of a water droplet in a case where the luminanceincreases from the center to the end, that is, the edge direction isdirected from the center to the end may also be employed.

Next, a detection process of the detection unit 25B will be describedwith reference to FIG. 32. FIG. 32 is a diagram illustrating a specificexample of the detection threshold value. Note that, in FIG. 32, theabscissa refers to the similarity value.

As described above, in the attachable matter detection apparatus 10Baccording to the fifth embodiment, the similarity with the template iscalculated as a value from “−1” to “+1.” Here, in a case where thesimilarity is close to “+1,” this means that the image is similar to thetemplate, that is, the camera image is similar to a water droplet inwhich the luminance increases from the end to the center.

In a case where the similarity is close to “−1,” this means that thecamera image is similar to a so-called negative/positive inversion ofthe template. That is, according to this embodiment, the templaterepresents a water droplet in which the luminance increases from thecenter to the end.

For this reason, as illustrated in FIG. 32, according to the fifthembodiment, the detection threshold value is provided in both thepositive and negative sides of the similarity. Specifically, asillustrated in FIG. 32, for example, a positive detection thresholdvalue is set to “+0.7,” and the negative detection threshold value isset to “−0.8.”

The detection unit 25B determines that a water droplet is attached tothe camera 2 in a case where the similarity value input from thematching unit 24B is equal to or higher than a positive detectionthreshold value, that is, equal to or higher than “+0.7,” or equal to orlower than a negative detection threshold value, that is, equal to orlower than “−0.8.”

In this manner, in the attachable matter detection apparatus 10Baccording to the fifth embodiment, the similarity value has positive andnegative detection threshold values. As a result, it is possible todetect both a water droplet having a luminance increasing from thecenter to the end and a water droplet having a luminance increasing fromthe end to the center through the matching process using a single typeof the template.

In other words, it is possible to detect various water droplets whilemaintaining a processing load. Note that the detection unit 25B maydetect a water droplet by using the processing described above inconjunction with FIGS. 26A to 26D together.

In the attachable matter detection apparatus 10B according to the fifthembodiment, the positive detection threshold value has an absolute valuesmaller than that of the negative detection threshold value as describedabove. This is because the negative similarity tends to produce a lot oferror detections of water droplets, compared to the positive similarity.

Each detection threshold value illustrated in FIG. 32 is just exemplary,and the present application is not limited thereto. For example, theabsolute values of the positive and negative detection threshold valuesmay be equal to each other. Alternatively, the absolute value of thepositive detection threshold value may be larger than the absolute valueof the negative detection threshold value.

Next, a processing sequence executed by the attachable matter detectionapparatus 10B according to the fifth embodiment will be described withreference to FIG. 33. FIG. 33 is a flowchart illustrating a processingsequence executed by the attachable matter detection apparatus 10Baccording to the fifth embodiment.

Here, Steps S401 and S402 will not be described because they have beendescribed in the fourth embodiment. The description will be givenstarting from Step S501 in FIG. 33.

The conversion unit 23B calculates a vector of an edge of each pixel onthe basis of the edge information input from the extraction unit 22(Step S501) and parameterizes each pixel on the basis of the edgedirection as described in conjunction with FIG. 30B (Step S502).

Subsequently, the matching unit 24B performs the matching processbetween the parameterized grayscale image and the template (Step S503).In addition, the detection unit 25B detects a water droplet on the basisof the detection threshold value of FIG. 32 (Step S504).

As described above, the attachable matter detection apparatus 10Baccording to the fifth embodiment can detect a water droplet with highaccuracy by using parameters of each R, G, and B of the twelve-colorwheel corresponding to the edge direction of each pixel.

The attachable matter detection apparatus 10B according to the fifthembodiment can detect both a water droplet having a bright center and awater droplet having a bright end through a single matching process.

Sixth Embodiment

Next, an attachable matter detection apparatus 10C according to a sixthembodiment will be described with reference to FIGS. 34 to 39. In theattachable matter detection apparatus 10C according to the sixthembodiment, each pixel is converted into codes depending on the edgedirection of each pixel, and a matching process is performed usingnormalized expression.

First, a configuration of the attachable matter detection apparatus 10Caccording to the sixth embodiment will be described with reference toFIG. 34. FIG. 34 is a block diagram illustrating an attachable matterdetection apparatus 10C according to the sixth embodiment.

As illustrated in FIG. 34, the attachable matter detection apparatus 10Caccording to the sixth embodiment includes a control unit 20C and amemory unit 30C. The control unit 20C has a conversion unit 23C, amatching unit 24C, and a detection unit 25C in place of the conversionunit 23, the matching unit 24, and the detection unit 25 illustrated inFIG. 21. In addition, the memory unit 30C stores code information 31C,template information 32C, and detection information 33C.

Note that the image acquirement unit 21 and the extraction unit 22 willnot be described here because they have been described in conjunctionwith FIGS. 2 and 28. The description will be made starting from theconversion unit 23C.

The conversion unit 23C calculates a vector of the edge of each pixel onthe basis of the edge intensities of the X-axis and Y-axis directions ofeach pixel input from the extraction unit 22 and encodes each edgedirection. This vector calculation method has been described inconjunction with FIG. 30A, and will not be described here.

The conversion unit 23C outputs a grayscale image obtained by encodingeach pixel to the matching unit 24C. Here, in the attachable matterdetection apparatus 10C according to the sixth embodiment, for example,a representative value of the edges of a plurality of pixels isobtained, and this representative value is encoded. This will bedescribed below in more details with reference to FIGS. 35A and 35B.

The matching unit 24C performs a matching process using normalizedexpressions of the encoded grayscale image input from the conversionunit 23C and a code pattern representing characteristics of a waterdroplet. Here, the normalized expression refers to a single codeexpressing a set of code strings.

Since the matching unit 24C performs the matching process using thenormalized expression, a cumbersome process such as the aforementionedsimilarity calculation is not necessary. For this reason, it is possibleto detect a water droplet while suppressing a processing load.

Note that the code pattern representing characteristics of a waterdroplet is stored in the template information 32C. In addition, thiscode pattern will be described below in more details with reference toFIG. 36A. Furthermore, the processing of the matching unit 24C will bedescribed below in more details with reference to FIG. 36B.

The detection unit 25C detects a water droplet attached to the camera 2on the basis of the code pattern extracted by the matching unit 24C.Note that the extraction process of the detection unit 25C will bedescribed below with reference to FIGS. 37 and 38.

Next, the encoding of the conversion unit 23C will be described withreference to FIGS. 35A and 35B. FIGS. 35A and 35B are diagramsillustrating the processing of the conversion unit 23C.

First, a pixel used to calculate the representative value will bedescribed with reference to FIG. 35A. Here, 8×8 pixels will be referredto as a cell, and 3×3 cells will be referred to as a block. In addition,the center cell of the block will be referred to as a main cell.

The conversion unit 23C creates a histogram representing the edgedirection and the edge intensity of each pixel for each block. Such ahistogram will be described with reference to FIG. 35B. Here, theconversion unit 23C derives the edge direction of the center coordinatesof the main cell from the histogram of the block.

If a representative value of the main cell of a single block is derived,the conversion unit 23C shifts the block by a single cell and creates ahistogram. Then, a representative value of the main cell of this blockis calculated.

That is, in the attachable matter detection apparatus 10C according tothe sixth embodiment, it is possible to reduce the data amount bycalculating representative values for each of a plurality of pixels. Forthis reason, it is possible to simplify the matching process of thematching unit 24C. Note that, since 8×8 cells are employed in theexample of FIGS. 35A and 35B, the data amount is reduced to 1/64.

Note that the numbers of pixels in the block and the cell of FIG. 35Aare just exemplary, and may be set arbitrarily. In this case, the numberof pixels in each cell may be changed depending on the size of the waterdroplet to be detected.

For example, in a case where it is desired to detect a small waterdroplet, the number of pixels in the cell is set to be small. In a casewhere it is desired to detect a large water droplet, the number ofpixels in the cell is set to be large. As a result, it is possible toeffectively detect a water droplet having a desired size.

The conversion unit 23C may simply create the histogram for each celland calculate the representative value of each cell on the basis of thishistogram. Note that the conversion unit 23C may encode overall pixelswithout calculating the representative value.

Next, the histogram will be described with reference to FIG. 35B. Notethat, in FIG. 35B, the ordinate refers to the edge intensity, and theabscissa refers to the grade of the edge direction. As illustrated inFIG. 35B, in the attachable matter detection apparatus 10C according tothe sixth embodiment, for example, the histogram is created byallocating the edge direction to each grade of 18 levels set to an angleof 20°.

Specifically, the conversion unit 23C creates the histogram of the blockby adding the edge intensity of each pixel of the block to the gradecorresponding to the edge direction. Subsequently, the conversion unit23C obtains a grade in which a sum of the edge intensity is maximizedfrom the created histogram.

In the example of FIG. 35B, a case where the grade of an angle of 80 to100° has a maximum value is illustrated. In this case, the conversionunit 23C sets a grade as the representative value if the sum of the edgeintensity is equal to or larger than the threshold value in this grade.

In the example of FIG. 35B, the sum of the edge intensity exceeds thethreshold value in the grade of “80 to 100°.” Therefore, theaforementioned condition is satisfied. For this reason, the grade of themain cell in this block is set to “80 to 100°.”

Subsequently, the conversion unit 23C converts the main cell into a codeallocated depending on the grade. Here, each of eighteen types of codes“0 to 9” and “A” to “H” is allocated to each grade. Note that “0 to 9”and “A to H” are codes allocated to each grade obtained by dividing from“0°” to “360°” in the unit of 20°. In addition, in a case where therepresentative value does not exceed the threshold value, that is, to acell having a low edge intensity, a code “Z” is allocated.

In this manner, the conversion unit 23C performs encoding for overallcells. As a result, in the encoded grayscale image, the codes arearranged in a grid shape. Note that the conversion unit 23C maycalculate the representative value using a statistic calculation methodother than the aforementioned calculation of the representative value.

In FIG. 35B, a case where the edge direction is classified into eighteengrades has been described. However, without limiting thereto, the numberof grades may increase or decrease from eighteen grades. FIG. 35Billustrates a case where the codes include “A” to “H” and “Z.” However,the codes may include other characters such as Hiragana or numericalvalues, symbols, and the like.

For example, in a case where a plurality of grades exceeding thethreshold value exist in a single block, the conversion unit 23C mayoutput the codes corresponding to the grades to the matching unit 24C inassociation with the grayscale image.

In other words, the conversion unit 23C may associate information on aplurality of edge directions with the grayscale image. In this case, thedata amount for detecting a water droplet increases. Therefore, it ispossible to detect a water droplet more accurately.

Next, the processing of the matching unit 24C according to the sixthembodiment will be described with reference to FIGS. 36A and 36B. FIG.36A is a schematic diagram illustrating an exemplary template accordingto the sixth embodiment. FIG. 36B is a diagram illustrating an exemplarymatching process using the matching unit 24C.

Note that, in FIG. 36A, in order to facilitate visual recognition, atemplate is schematically illustrated using actual edge directionsinstead of the aforementioned codes. As illustrated in FIG. 36A, in theattachable matter detection apparatus 10C according to the sixthembodiment, the template has a code pattern as a code stringrepresenting characteristics of a water droplet. Specifically, thetemplate includes, for example, an upper side pattern, a lower sidepattern, a left side pattern, and a right side pattern.

Here, each side pattern illustrated in FIG. 36A indicates each side of arectangular that internally or externally envelops a water droplet. Inaddition, in FIG. 36A, a case where each edge direction of each sidepattern is directed to the center is illustrated. In this case, theluminance of the water droplet increases from the end to the center.That is, the center is bright, and the end is dark as a characteristicof the water droplet.

Note that, in the attachable matter detection apparatus 10C according tothe sixth embodiment, the luminance of the water droplet increases fromthe center to the end. That is, each side pattern may represent acharacteristic of a water droplet in which the center is dark, and theend is bright. As a result, it is possible to detect various waterdroplets.

Note that, although four patterns including upper, lower, left, andright side patterns are exemplified in FIG. 36A, a pattern having aslope direction may also be employed. As a result, it is possible toimprove the water droplet detection accuracy.

The code string indicating the characteristics of the water droplet maybe, for example, an array of codes arranged in an arc shape. Inaddition, the matching unit 24C may restrict the region for performingthe normalized expression depending on each side pattern.

A matching process of the matching unit 24C will be described withreference to FIG. 36B. Note that, here, for convenient descriptionpurposes, the upper side pattern of FIG. 36A is indicated by the codes Ato F. In addition, in (a) and (b) of FIG. 36B, a part of the grayscaleimage encoded by the conversion unit 23C is schematically illustrated.

As illustrated in (a) of FIG. 36B, if the code patterns are alignedsequentially in order of A to F, the matching unit 24C determines thatthis code pattern matches the upper side pattern.

Specifically, as illustrated in (a) of FIG. 36B, for example, if anarray in which “A” is repeated three times, “B,” “C,” “D,” and “E” arerepeated twice, and “F” is repeated three times satisfies an arrangementsequence of each code of the upper side pattern, the matching unit 24Cextracts this array as the upper side pattern.

This is because the repetition frequency of the code is differentdepending on the size of the water droplet. That is, as the size of thewater droplet increases, the length of each code string increases. Inthis manner, by allowing repetition of the codes, it is possible toextract a code string indicating a plurality of water droplets havingdifferent sizes through a single matching process.

Therefore, it is possible to detect a water droplet while reducing aprocessing load. Note that a plurality of patterns of the code stringshaving different lengths depending on the size of the water droplet maybe prepared for each side, and the matching unit 24C may extract thecode strings using all of the patterns.

Since the water droplet typically has a spherical shape, the repetitionfrequency of each code becomes linearly symmetric with respect to thecenter. For this reason, the matching unit 24C excludes a code stringhaving imbalance from the extracted code strings.

Specifically, as illustrated in (b) of FIG. 36B, the matching unit 24Ccarefully investigates balance between “A” and “F” located in both ends.Here, in FIG. 36B, “A” is repeated three times, and “F” is repeated tentimes.

In this case, in a case where the number of “A” and the number of “F”are different twice or more, the matching unit 24C excludes this codestring pattern even it satisfies the arrangement sequence. As a result,it is possible to prevent erroneous extraction of an unnecessary codepattern other than a water droplet and suppress erroneous detection of awater droplet.

For example, in a case where the extracted code string is longer than athreshold value, the matching unit 24C may exclude this code string fromthe matching. This is because, if the code string is long, a possibilityof the water droplet is low. For this reason, it is possible to suppresserroneous detection of a water droplet. Note that it is assumed thatthis threshold value is derived as an optimum value through a statisticmethod or the like in advance.

Next, a detection process using the detection unit 25C according to thesixth embodiment will be described with reference to FIG. 37. FIG. 37 isa diagram illustrating a detection process using the detection unit 25Caccording to the sixth embodiment. Note that, similarly to FIG. 36A,FIG. 37 schematically illustrates actual edge directions instead ofcodes.

Here, a case where an upper side pattern is initially extracted by thematching unit 24C will be described. First, the detection unit 25C setsa substantially rectangular detection region R1 on the basis of a widthof the upper side pattern.

Subsequently, it is assumed that the matching unit 24C extracts theright side pattern in a position deviated from the detection region R1.In this case, if a central coordinate of the detection region R2 of theright side pattern is within the detection region R1, the detection unit25C performs a process of integrating both the detection regions R1 andR2.

Then, for example, in a case where the lower side pattern or the leftside pattern is extracted from the integrated detection region R3, thedetection unit 25C detects a water droplet in the integrated detectionregion R3. In other words, the detection unit 25C detects a waterdroplet by setting a detection condition in which a pattern indicatingeach side having different three or more directions is extracted in thedetection region R3 (hereinafter, referred to as a directionalcondition).

Note that, instead of this directional condition, the detection unit 25Cmay set a water droplet detection condition (hereinafter, referred to asa frequency condition), for example, in which a pattern indicating eachside is extracted frequently by a predetermined number or more (forexample, four times including upper, lower, left, and right sides) inthe integrated detection region R3.

In this manner, by setting the directional condition including three ormore directions or the frequency condition as the detection condition, awater droplet is detected even when all of the upper, lower, left, andright sides are not extracted. That is, it is possible to detect, forexample, a semicircular water droplet removed from the camera image.

Note that the directional condition may be changed, for example,depending on a region for detecting a water droplet. For example, in acenter region of the camera image, the directional condition is set infour directions. As a result, it is possible to improve the waterdroplet detection accuracy.

The directional condition is set to “twice” in the regions of fourcorners of the camera image. As a result, it is possible to detect apartially cut fan-shaped water droplet appearing in four corners of thecamera image.

Note that, although a case where the detection regions are integratedwhen the central coordinate of the detection region R2 is settled in thedetection region R1 of the upper side pattern has been described hasbeen described in FIG. 37, the present application is not limitedthereto. That is, if at least parts of the detection regions R1 and R2are overlapped, both detection regions may be integrated.

The integrated detection region R3 may be a logical product between thedetection regions R1 and R2 or may be a logical sum of the detectionregions. In addition, although the detection regions R1 and R2 have arectangular shape in FIG. 37, the detection region may have any othershape such as a circular shape without limiting thereto.

Note that, as described above in conjunction with FIG. 26D, thedetection unit 25C may detect a water droplet on the basis of adetection result from a plurality of frames.

Next, an exclusion process of the detection condition using thedetection unit 25C will be described with reference to FIG. 38. FIG. 38is a diagram illustrating a condition for excluding a pattern on thebasis of the detection condition. In FIG. 38, a part of the grayscaleimage is enlarged, and the edge directions are schematicallyillustrated.

Note that, in FIG. 38, the length a1 is set to, for example, 1.5 timesthe length a2. That is, the length a2 is a two thirds (⅔) of the lengtha1. Here, the left side pattern tends to be easily extracted, forexample, from the left two-thirds region of the grayscale image (theleft region with respect to the dotted line in FIG. 38) even when awater droplet is not attached to the camera 2.

For this reason, in a case where a plurality of left side patterns areextracted, and a pattern representing other sides is not extracted inthis region, the detection unit 25C exceptionally excludes the patternfrom the detection target even when the aforementioned frequencycondition is satisfied.

As a result, it is possible to suppress erroneous detection of a waterdroplet. Note that, although the left side pattern has been exemplifiedhere, this similarly applies to other side patterns.

Next, a processing sequence executed by the attachable matter detectionapparatus 10C according to the sixth embodiment will be described withreference to FIG. 39. FIG. 39 is a flowchart illustrating a processingsequence executed by the attachable matter detection apparatus 10Caccording to the sixth embodiment. Here, since Steps S401 and S402 havealready been described, the description will be made from Step S601 inFIG. 39.

First, the conversion unit 23C calculates a representative value bycreating a histogram on the basis of edge information extracted by theextraction unit 22 as illustrated in FIGS. 35A and 35B (Step S601).Then, the conversion unit 23C performs an encoding process for encodinga representative value of each cell (Step S602).

Subsequently, the matching unit 24C performs a matching process for theencoded grayscale image using a normalized expression (Step S603). Inaddition, the detection unit 25C detects a water droplet as illustratedin FIG. 37 (Step S604).

As described above, in the attachable matter detection apparatus 10Caccording to the sixth embodiment, each pixel is encoded, and thematching process is performed using the normalized expression. As aresult, it is possible to simplify the matching process. That is, it ispossible to detect a water droplet with high accuracy while suppressinga processing load.

In the attachable matter detection apparatus 10C according to the sixthembodiment, it is possible to improve detection accuracy of waterdroplets having different sizes or a water droplet partly cut from thecamera image by using the normalized expression in the matching process.

Note that, the attachable matter detection apparatuses 10A, 10B, and 10Caccording to the fourth to sixth embodiments may be appropriatelycombined. For example, the method of calculating the representativevalue of FIG. 35A may be employed in the fourth and fifth embodiments.

In the attachable matter detection apparatuses 10A, 10B, and 10Caccording to the fourth to sixth embodiments, a frame interval forobtaining a camera image from the camera 2 may be changed depending onthe purpose of detecting the water droplet. For example, it is necessaryto detect a water droplet as soon as possible in a case where the cameraimage is presented to a driver during a backward movement of the vehicleC.

For this reason, in such as case, all of the frames photographed by thecamera 2 are acquired, and a water droplet is detected from all of theseframes. Meanwhile, for example, in a case where the detection purpose isthe sensing in automatic parking and the like, the camera image may beacquired, for example, at every several frames.

In this case, the cameras 2 that acquires camera images may be changedon a frame-by-frame basis, such as rear camera 2-2→front camera2-1→right-side camera 2-3→left-side camera 2-4.

In the attachable matter detection apparatuses 10A, 10B, and 10Caccording to the fourth to sixth embodiments, the water dropletdetection process may be performed by changing a resolution of thecamera image. For example, in a case where water droplet detection isperformed by lowering a resolution, it is possible to reduce aprocessing load of the detection process. Note that the resolution maybe changed depending on a purpose of the water droplet detection.

In the fourth to sixth embodiments, a case where the attachable matterdetection apparatuses 10A, 10B, and 10C extract a gradient of theluminance in each pixel of the camera image L as the edge informationhas been described. However, the present application is not limitedthereto.

The attachable matter detection apparatuses 10A, 10B, and 10C mayextract a gradient of saturation in each pixel of the camera image L asedge information and detect a water droplet attached to the camera 2 onthe basis of such edge information. In this case, the attachable matterdetection apparatuses 10A, 10B, and 10C can accurately detect a muddywater droplet mixed with mud, sand, and the like attached to the camera2.

Specifically, for example, assuming that the HSV color space isemployed, the extraction unit 22 of the attachable matter detectionapparatuses 10A, 10B, and 10C may extract the saturation on the basis ofa formula “saturation (S)=(Imax−Imin)/Imax,” where “Imax” denotes amaximum value of R, G, and B of each pixel of the camera image L, and“Imin” denotes a minimum value.

Assuming that the HSL color space is employed, the extraction unit 22may extract the saturation on the basis of a formula “saturation(S)=(Imax−Imin)/(Imax+Imin) if L≤0.5” or “saturation(S)=(Imax−Imin)/(2−Imax−Imin) if L>0.5” and a formula “brightness(L)=(Imax+Imin)/2.”

Subsequently, the conversion unit 23 according to the fourth embodimentcalculates a value obtained by squaring the saturations in X-axis andY-axis directions and adding them as a saturation-based edge amount ofeach pixel. In addition, the conversion unit 23 may binarize each pixelof the camera image L by comparing this edge amount and the binarizationthreshold value THa as illustrated in FIG. 22. Note that thisbinarization threshold value THa optimized for the saturation may beemployed.

Then, the attachable matter detection apparatus 10A performs theprocessing already described above using the matching unit 24 and thedetection unit 25 so that it is possible to accurately detect a muddywater droplet attached to the camera 2.

The conversion unit 23B according to the fifth embodiment may calculatethe edge directions based on the saturation of each pixel andparameterize each pixel by allocating three-dimensional parameters ofFIGS. 30A and 30B in the edge directions.

Subsequently, the matching unit 24B performs a matching process usingthe template of FIG. 31. In the case of a muddy water droplet, thesaturation increases toward the center. For this reason, the detectionunit 25B can detect a muddy water droplet only when the value of thezero-mean normalized cross-correlation calculated by the matching unit24B exceeds the positive detection threshold value of FIG. 32.

That is, the attachable matter detection apparatus 10B according to thefifth embodiment can suppress erroneous detection of a muddy waterdroplet by setting only the positive detection threshold value when themuddy water droplet is detected.

The conversion unit 23C according to the sixth embodiment encodes eachpixel of the camera image L with respect to the direction of thesaturation gradient as illustrated in FIGS. 35A and 35B. In addition,the matching unit 24C extracts a code string representing a waterdroplet through a matching process using the normalized expression.

The detection unit 25C detects a muddy water droplet attached to thecamera 2 on the basis of a code string extracted by the matching unit24C. In this case, as described above, the saturation increases towardthe center in the case of the muddy water droplet.

For this reason, the detection unit 25C can accurately detect a muddywater droplet by detecting a code string pattern having the saturationincreasing toward the center on the basis of the code string extractedby the matching unit 24C.

In this manner, the attachable matter detection apparatuses 10A, 10B,and 10C can accurately detect a muddy water droplet by using thesaturation instead of the luminance of each pixel as the edgeinformation. Note that the attachable matter detection apparatuses 10A,10B, and 10C may simultaneously detect, for example, both a waterdroplet having the luminance increasing toward the center and a muddywater droplet having the saturation increasing toward the center througha single matching process.

Although a case where all of the attachable matter detection apparatuses10, 10A, 10B, and 10C and the attachable matter removal system 1 areapplied to an in-vehicle camera 2 has been described in each ofaforementioned embodiments, the above-described embodiments may also beapplied to other types of cameras such as a monitor/security camera setinside or outside a building or a street.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An attachable matter detection apparatuscomprising: a memory that stores therein condition information; and aprocessor operatively connected to the memory, the processor beingprogrammed to: cause an imaging device, which is mounted on a vehicle,to capture a photographic image of a circumference of the vehicle, theimaging device including an image sensor; acquire, as a determinationtarget area, a detection area of the photographic image in which anattachable matter to the imaging device is estimated to be present byexecuting a predetermined detection algorithm on the photographic imagecaptured by the imaging device; create, for the acquired determinationtarget area, a histogram of an edge intensity, a histogram of luminance,and a histogram of saturation, each of the histograms including aplurality of grades, and each of the grades including a correspondingfrequency, wherein the condition information stored in the memoryincludes an exclusion condition obtained by combining (i) ratios betweenthe grades of the edge intensity, (ii) ratios between the grades of theluminance, and (iii) ratios between the grades of the saturation, theexclusion condition not being suitable for characteristics of anattachable matter to the imaging device; compare, with the conditioninformation stored in the memory, a combination of ratios between thegrades of the edge intensity, the luminance, and the saturation of theacquired determination target area; when the combination satisfies theexclusion condition, determine that the attachable matter does not existon the imaging device corresponding to the determination target area;and when the combination does not satisfy the exclusion condition,determine that the attachable matter exists on the imaging devicecorresponding to the determination target area.
 2. The attachable matterdetection apparatus according to claim 1, wherein each of the createdhistograms includes at least three grades including “low,” “middle,” and“high”.
 3. The attachable matter detection apparatus according to claim1, wherein the processor is further programmed to: determine whether ornot the attachable matter exists based on a change amount of at leastone of the created histograms between a current frame and a previousframe.
 4. The attachable matter detection apparatus according to claim3, wherein the processor is further programmed to: acquire each of aplurality of partitioned areas set for the photographic image as thedetermination target area; and determine that the attachable matterexists with respect to a partitioned area in which the change amountsatisfies a predetermined detection condition.
 5. The attachable matterdetection apparatus according to claim 4, wherein the detectioncondition is able to be set for each partitioned area.
 6. The attachablematter detection apparatus according to claim 4, wherein the processoris further programmed to: adjust the detection condition when apredetermined trigger that is suitable for adjustment of the detectioncondition is generated; adjust the detection condition to be reinforcedwhen the determination target area is positioned in a lower region ofthe photographic image.
 7. The attachable matter detection apparatusaccording to claim 1, wherein the processor is further programmed to:enlarge or reduce a size of the determination target area to match areference size when the histograms are created.
 8. An attachable matterdetection method comprising: causing an imaging device, which is mountedon a vehicle, to capture a photographic image of a circumference of thevehicle, the imaging device including an image sensor; acquiring, as adetermination target area, a detection area of the photographic image inwhich an attachable matter to the imaging device is estimated to bepresent by executing a predetermined detection algorithm on thephotographic image captured by the imaging device; creating, for theacquired determination target area, a histogram of an edge intensity, ahistogram of luminance, and a histogram of saturation, each of thehistograms including a plurality of grades, and each of the gradesincluding a corresponding frequency; compare, with condition informationstored in a memory, a combination of ratios between the grades of theedge intensity, the luminance, and the saturation of the acquireddetermination target area, wherein the condition information stored inthe memory includes an exclusion condition obtained by combining (i)ratios between the grades of the edge intensity, (ii) ratios between thegrades of the luminance, and (iii) ratios between the grades of thesaturation, the exclusion condition not being suitable forcharacteristics of an attachable matter to the imaging device; when thecombination satisfies the exclusion condition, determining that theattachable matter does not exist on the imaging device corresponding tothe determination target area; and when the combination does not satisfythe exclusion condition, determining that the attachable matter existson the imaging device corresponding to the determination target area.